Proven Models that Help GTM and Product-led Teams Scale | CircleCI CEO Jim Rose

Awesome. Thanks Taylor. Hello to everyone. Hello virtually. Hope everyone is happy and well. My name is Jim Rose. I'm the CEO of CircleCI. Just here to share some of the lessons that we've had over the last seven, almost eight years of making a transition from a subscription based model into a usage based business and some of the key lessons that we've learned along the way and happy to share those as we go. Sorry. So what are some of the successful building blocks for a usage based world? It's really three themes that we're going to talk about during the course of this presentation. First is being able to match the value that you deliver as a platform, ultimately with how you price and package and sell to your customer. The second is really changing the relationship with your customer from a consumer to, to being your partner in growth. And then the third is making everyone, both internally and externally responsible for the success of the relationship between you, the platform and you, the vendor and the customer. And so before jumping into all of that though, I think it's always useful to gather some context. There's no good generic startup feedback and so it's always nice to be able to get an idea of where we're coming from and what some of our biases are so you can understand some of the feedback that we're going to give and you can determine whether or not it actually applies to your business. So a quick two minute primer on what CircleCI is. So what is CircleCI? We are a continuous delivery platform. So we help software delivery teams ultimately build, test and ship changes to their application. So something changes upstream. You want to be able to build it, test it, make sure it's good before you ship it downstream into the hands of your downstream consumer. And we are deeply, deeply in the technical stack. We sell to software developers and software delivery teams and we are a cloud based service. We offer this predominantly as a multi tenant service where customers come to circleci.com, they sign up, they use the service and go on. And as anyone who's had to sell to developers in the past knows, developers are a pretty persnickety audience. They are very specific about the tools that they like to use because they're craftspeople. They love to be able to use the best tools that they have at their disposal and they want to be able to test it before they commit to it. And that idea of trying to drive trial and getting customers accustomed to and developers accustomed to how your tool works is really the key cornerstone of a go to market motion for any developer tool. Now, there's a collection of folks out there that do that through open source distribution. They create an open source package. You can download it, you can play with it, you can use it for free, you can integrate it into your package. But because of what we do, we offer our service as a freemium based service so that our customers and our developers can come to CircleCI, they can sign up, they can use it for free, and then ultimately they go along a customer adoption journey with us. And that initially starts with that freemium engagement where the customer comes on board, they can integrate CircleCI into their project. As long as they're building open source, they're trialing, or they're just doing side projects, they can build on our platform indefinitely for free. We're happy to support them as much as they want to build. Eventually though, if you start to build software commercially, what we seek to do then is get into more of a commercial relationship. So eventually your freemium plan hits a cap. If you need more services out of the CircleCI platform, you have to then convert into a paid account. More often than not, that's through a self serve credit card swipe. At that point you have access to a paid plan and you can use more of it. And eventually, as your usage of circleci grows, if it gets the size where it becomes a situation where you have a substantial amount of spend, we seek to get into a committed relationship and a direct selling relationship where it's good for the customer in helping us understand what they're trying to use, how often they're trying to use our platform, and what they need from the platform. But it's also good for us because the more we know about what customers are trying to achieve, the better we can plan as a platform. So it's a pretty traditional bottoms up motion and it's again entirely cloud based. So it's a freemium based approach as opposed to open source. And one really important point is that as we sell to individual contributors, eventually those individual contributors have to ultimately convince decision makers and buyers that the services that they've been using in the past that no one was aware of are things that we should be using in the future. So our sales process tends to be really multi leveled as we continue to grow with the customer. And when we started out selling CircleCI to developers, it really started out as a very tightly defined and narrow target. So CircleCI started and was launched in early 2012. And when we went to market, we had a very traditional SaaS subscription model. We sold one version of the platform and it was called a container. And a container cost $50 a month. If you needed more containers, you could pay more for more containers. But every container was the same, every container cost $50 a month. And the model worked because every customer that we had was seeking to buy the exact same thing, which was this very basic Linux container and set of resources that they would purchase from us. But as we grew as a platform and as our customers grew as a platform and their utilization of our platform increased, it got substantially more diverse. So what used to be a very tightly defined. Whoops. A very tightly defined offering where it was one particular type of container that we sold over and over and over again as people came with different requests, different size containers, different operating systems, different locations. All of a sudden what used to be a very tightly defined pricing model and packaging model where one size ruled them all, became something that was no longer supportable. We had all of these requests coming in for things that no longer fit within our pricing and packaging model. So about five years into our existence, we chose to actually change the way that we went to market and how we sold to market away from a traditional SaaS subscription model into a usage based model. And in a usage based model, in our particular case, we created essentially a creditized model. If you're in the Bay Area, think of it a little bit like a Clipper card. A customer opens an account with us, they deposit money into that account, that money gets creditized, and then depending on how they build on our platform, they consume those credits over time. And those credits get consumed by the number of users that are on our platform, how they're supported, and then ultimately the types of resources that they consume. And as they're working through that consumption, that balance comes down. And when the balance gets to zero or gets close to zero, we then reload the account with more credit. But it's entirely a usage based model and no longer a subscription based model. And that happens to fit very well for our particular business because in software delivery and in software testing, demand tends to be very uneven. Sometimes you need a lot of circleci because you're busy trying to get a new release out to market. Other times you need none. It might be the holidays and no one's actually building. And very often you have no idea what you need to buy from the Circleci platform until you're actually trying to solve the problem. So you may not know that you need macOS resources or Windows resources or Linux resources until a developer is actually trying to build that service. So you're really not clear about what you're trying to buy ahead of time and accreditize. And a usage based model allows for that variability for the buyer so that they don't have to have that kind of visibility into what their customers are trying to do. So in the first five years we made this change to a usage based model. And when we made the change, a lot of people just said, well, that's interesting. That's just the different way that you're charging a customer. How different does that actually make your business? And what we have realized over the last five years is that this change from a traditional SaaS based business into a usage based business fundamentally alters the some of the core relationships and some of the core processes that you have inside of your business. So context, we're cloud based, we're bottoms up, we're freemium, not open source, and we're usage based. Now what are the impacts of that usage based evolution? What actually changes when you as a customer and you as a platform change the way that you're charging your users? And the important thing to note is that usage based businesses and usage based economics are not the same as traditional subscription or traditional SaaS based economics. And it differs in one very important respect. And that is how you generate revenue. So how is revenue generated in a usage based business? And how is that different from a subscription based business? So use an example of a health club. So I'm here in our office in San Francisco, there's an Equinox down the street. For $200 a month or something like that, I can join Equinox. And for that $200, I can then go and use that gym as many times as I want. And as the gym in a traditional subscription base and membership model, I make $200 regardless of how many times the customer comes in to use the gym. Whether they use it once or they use it a hundred times, it's the same revenue to me and I get to book it evenly and every month. And that's your traditional SaaS based subscription model. But when you get into usage based models, the way you generate revenue changes. So take that same $200 fee instead of it being a subscription and a monthly membership into the club. What if it was a deposit and I made a $200 deposit into my health club and the only way I work off that deposit and the gym gets to recognize it as revenue is if I actually come in and use the gym. I come in, take classes, I go swim in the pool, I go lift weights, I go for A run, whatever it happens to be. If in this, in a usage based model, the only time that I get to recognize that $200 as revenue is when people come in and actually use a service. So in a subscription based model, the problem is the customers, right? I've already paid how much you use. It is the customer's problem. They need to determine how often they're going to come into the gym. In a usage based model, it's the platform's problem. It's the gym's problem. Just because I have your money doesn't mean I can actually recognize it in revenue. I have to build something that's so enticing and so inviting for the user that they actually want to come in and use it and use it more and more often over time. And that fundamental change changes the relationship between you and the customer. I as a platform need to ensure that everything that I build, everything that I do, all the support that I provide, gets you to use our service and use our service continuously. And that is a very big difference between usage based models and subscription based models. In subscription based model, you start out with a pretty understood pattern. Your builders build a service, your sellers sell that service, your customers buy the service and hopefully they consume the service. But in a usage based model, because the sale is only the beginning of the relationship, it's only the initial deposit, not the actual revenue. That entire pattern, instead of it being sort of unidirectional from builder to customer, it it ultimately gets all mixed up. Builders have to be really clear about what customers are actually going to use, not what they say they want. Sellers need to be very clear about what customers are actually using and how to generate the right commitments and the right usage from those customers. And then the sellers and the builders ultimately need to be in alignment as well. And what we have found is that over time, usage based models allow for businesses like ours to stay really tightly aligned against the value that the, that the customer wants out of our platform. The customer wants to use our platform to be able to build software and build software. Often, however, sometimes the existing processes and the existing models that customers have in place aren't really built for this new evolving type of relationship between the platform and the customer. And so what I wanted to do is pull out three specific examples where we see this relationship between the customer and the platform begin to break down and some of the pathways and some of the lessons learned and some of the strategies that we have used to break it. Situation number one, when you realize that you have a mismatch between the needs of a usage based platform and the understanding of the customer is around this idea of feature requests. We've all had these. These situations arise when a salesperson comes in. They say, we have this amazing big customer who's really interested in buying our service. They're going to spend millions of dollars with us and they want to do that. They want to sign the contract, but they need a certain number of features to be able to get the contract signed. And when you're on the builder side and you're thinking about it from a product and from an engineering and delivery perspective, you're like, well, sure, we can build anything, but we can't build everything. And of course, the things that we do build, we want to make sure that our customers actually use. Great, all right, great. The customer is ready to go. They're really interested in using the service. Then what happens? Next thing you know, you get the feature request list. I can think about this because I actually just received one of these that's 24 pages long where the customer has 123 different requests about the service that they want to buy from from us as the platform. Now, granted, they've never actually used the service at that scale. So these 123 requests, you're not even 100% sure that they know what they want, but they tell you absolutely every single one of these is a must have. If we don't get it, we can't close the deal. And oh, by the way, it's not just that we need all of these features and we need them like to be able to close a deal, but we need them by a certain date as well. It all has to be done by December 3rd. Why December 3rd? Nobody knows. It's usually because someone has a meeting, somebody has a bonus or an MBO tied to a certain delivery on a certain date, or maybe somebody's going on vacation and they want to get it done before they actually go out on vacation. But regardless, there's this idea of it has to be done by this date and we need all of these things done before we can actually sign the deal. Now, we've all been in a situation where we've had to do these deals where it's big customer X. Yes. That doesn't seem like exactly the right relationship and it doesn't seem like the exact right deal, but you do it for other various and a sundry reasons. Maybe it's the deal in the logo that ultimately gets you to five other deals. Or maybe it gets you to a financing Round or maybe it tides you over from a cash perspective for another quarter while you sort of figure out how to get to profitability. But we've all been in situations where we've had to do these deals and these deals are typically reflective of a misunderstanding of a usage based business that's trying to deliver value to the customer that the customer actually uses versus what the customer is trying to buy. Modern software delivery and modern application delivery doesn't really work all that well on drop dead days. It usually the way I think about modern software delivery is it's really about sort of a moving picture. It's like a film. It's a film that I can jump into at any one given time, but the film continues, it progresses, it continues and evolves as a story over time. For example, at CircleCI, we release changes to our application anywhere between 700 to 1,000 times a week. There is no one fixed version or one fixed vision of what the application is. But when you start selling to folks that are accustomed to buy buying in more traditional SaaS models, they don't think about the platform as this is, this moving picture as this ever evolving story. They think about it in terms of snapshots, which is, I'm buying an application that looks like this. Exactly at this moment in time, I'm buying version 14. It has these services, these features, and I need to know that with 1414 that all of those things are going to be available. And so now your delivery model and your purchasing model and your customer expectation model no longer match up and the question comes down to do you close the deal or not? And again, we've all closed these deals for various and essential reasons. And sometimes it's the right thing to do from a business perspective. But when you get into these usage based models, oftentimes it's not the right thing to do from a product perspective because more often than not you've run into a situation where you've built those 123 features for that user, those 123 features that were absolutely, positively must have and had to be in place for them to be able to purchase from you. But when you go back and you look several months later, you realize, huh, none of those features are actually being used. And when you go and talk to the customer and you go and talk to the seller, you're like, well, you said that all of these things were absolutely critical and important to be able to build. And oftentimes the customer is no longer there. They've moved on to different projects and different Opportunities, the seller and the go to market team that's responsible for supporting that customer oftentimes doesn't actually know what's causing the kinks in adoption either. And In a traditional SaaS model, this doesn't leave as much of a mark because again, you make the same amount of money regardless of how much the customer uses the platform. So again, using the health club analogy, I get the $200 per month regardless of how many times you use the gym. But in a usage based business, I only make money if you use the platform. If you don't use the platform, I don't get to recognize those deposits as revenue. And even worse, if I built a number of features for you as a customer that you aren't using, it's not just that I wasted time building those features that aren't being used, is that the opportunity cost is that I didn't build a number of features that probably would be used in other contexts. So it's very much a double whammy. And how do you get out of this box? How do you get out of this situation where your interest is as a platform while being directly aligned with the customer, aren't aligned with how the customer buys? And the strategy that we've adopted is basically building in the open. So publishing a public roadmap, opening up idea portals, having discussion boards and forums for your customers, having customer advisory boards, having customer events where they have input into the product roadmap. And the idea is that the product roadmap is the movie, it is the film that we all agree that we're making and that we're building and the future that we're building towards. But customers have the ability to influence the ending of that. They have the ability to say we need X, Y and Z, because these are going to be keys to us unlocking usage. And we'd like to provide that feedback to be able to adjust how you think about your own public roadmap and where you're actually taking the product. Now, building in public and building in the open is really uncomfortable for most software delivery teams and for most product teams, it opens you up, you're very vulnerable. Customers have the ability to see what you're building versus what you're not building. And oftentimes what you're not building, many customers think is very important and they can get pretty emotional about that. Your competitors can see what you're building, your internal constituents can also see what you're building. But at the end of the day, it's a trade off. And the trade off is by providing transparency and exposure. Ideally, you reduce these wasted cycles on features that don't matter and, and don't ultimately drive usage and revenue for you as a business. So we always put forth as much as possible build in the open so your customers are knowing where you know where you're going and then can influence that as you go. The next situation that we oftentimes run into is uptime. So same deal, same big customer comes in. Uh oh, I need 123 features. But not only do I need 123 features, but I need four nines uptime and I need you to, to memorialize it in writing in an sla. And if you don't meet these requirements, then I have to have some kind of out in the contract or some kind of clawback. And when you're a builder and you think about that, you're like, well, can you describe to me what actually four nines uptime uptime means? Because when we're building and we build software, oftentimes we only control a small component of what's being constructed. You're usually integrated with a whole mess of different services, different APIs and different inputs and outputs into the process. And from a seller's perspective, oftentimes you have no idea and the customer doesn't know actually what an SLA is. And so it falls on the builders to figure out how to answer that and how to address it. And usually what you come up with is something that's technically accurate but is ultimately a terrible compromise. And so if you ever go out and look at SaaS status pages, what you will find is that everyone's platform is up. Five nines, it's always up. Everything is always running, everything is always green. If you're in the world where you actually have to call on those services and integrate them together, what you quickly realize is that is totally not true is that in many cases those services go up and down and they, they're available and unavailable. But the answer when people ask you for SLAs is you try to come up with something that's technically accurate but at the same time is enforceable. And so I picked a couple of examples of status pages from partners that we have and you realize that the technically accurate answer is to tell the customer is part of my infrastructure available and can you access it. So, for example, in this particular SaaS example, I can tell you that database Shard 7 in the Americas is available when I ping it. I can't tell you whether or not it's actually going to serve any value and support your application over time. So it's both bad from a, from a platform perspective because we're presenting uptime stats that we are busy trying to maintain but don't actually mean anything. And it's bad from a customer's perspective because we're presenting stats that ultimately show that things are up, but you may or may not actually get value out of the service cause you may not be able to solve the problem. And so the solution that we've tried to put in place is instead of thinking about uptime from the perspective of operational metrics, instead what we've tried to think about this through is through the lens of availability, through the lens of what is the customer seeking to accomplish and for that particular job to be done, can it be accomplished? In our particular case, that means can a customer actually build software? And can they build software in a Linux machine? Can they build it in a Mac OS machine? Can they build it in other environments? That at the end of the day is what customers are purchasing from us. Does that necessarily mean that the payment pages are available? 4 or 5 nines? Maybe or maybe not. Does that mean that the authentication or the identity layer is always available? Maybe or maybe not. But the thing that the customer buys from us is ultimately the ability to buy software. And if you think about most of the services that we use, we usually buy it for one, maybe two, maybe three specific use cases of I use Spotify because I want to listen to music. I use Netflix because I want to watch videos. I use Slack because I want to be able to direct message my colleagues and potentially my customers. But there are very specific use cases and very specific jobs that need to be done. And building availability around those jobs and the SLAs specifically around the availability of those key metrics is going to get both you as the platform and the customer into a better spot. The customer is going to know that the thing that they need to get done as possible and you as a platform are going to know where to spend your valuable and very limited engineering resources to make sure that the things that matter are ultimately bulletproof and available. And the example I would use here is it's like a plane. You think about, I live in San Francisco and I'm in San Francisco today. To go from San Francisco to New York, if I want to take a plane, that's an incredibly complex event that has to occur. It involves human resources, coordination in the skies, big machines, airports, etc. But at the end of the day, the contract between myself and the airline is, if I bought a ticket, can I get a seat? Is the plane running on time? And oh, by the way, does it have WI fi? That works at least half the time as I'm flying coast to coast so that I can pretend like I'm working and then stop doing that and watch a Marvel movie. If it meets all of those requirements, then it's conceptualized from the perspective of the customer and that contract and the understanding is clear and understood. And again, using the example of an airline in an airplane, you know when that's been broken. So if anyone has flown to New York recently, you know that while you might leave San Francisco on time and you might technically land in the Newark airport on time, you might end up sitting on the tarmac for an hour waiting for a gate. And while the airline considers that to be meeting their own SLA and meeting their own uptime requirements, from a customer perspective, that doesn't actually feel right. So what we always advocate is pick the jobs that matter, put those in front of the customer, get clarity on exactly what uptime means for those jobs, and then spend all of your time and resources in making sure that that actually happens and is always available. So the last thing I wanted to talk about is situation number three, which is ultimately who's responsible for success. So if you think back to the health club model and you think back to a traditional SaaS subscription model, when you sell, when a customer actually buys something from the platform, I get the revenue, it's just a question of when I get to recognize the revenue. And so the sales process from a platform perspective is ultimately the end point or the sort of the beginning of that success relationship from a customer's perspective. Ideally you want to be able to use the platform. When you get into usage based models, usage based models don't look that way. When a customer purchases from our platform, they've made a deposit and that deposit is great. That deposit is indicative of a relationship and a commitment between them as customers and ourselves as a platform. But doesn't necessarily mean that I get to recognize any revenue. That initial deposit is the beginning of the relationship. It's, I now need to do everything in my power to make sure that the customer is getting maximum value out of the platform, because when they get maximum value out of the platform, we get to recognize that as revenue. And so that that interest or those interests and that relationship is aligned, but it doesn't stop. And so I know our go to market teams are always, they're always upset when I ask who's responsible for success, who's responsible for owning usage? And the answer is, ultimately everyone is. Everyone is responsible for driving success and usage of the customer. From a builder's perspective, when you think about driving success in the usage of the platform, it comes down to innovation velocity and being able to innovate as quickly as possible with the features that matter the most and ultimately are used the most by the customer and making sure that they're available. So making sure that the things that we're building count, that the customers actually use it and that the customers can use it, and getting yourselves aligned from a builder perspective around those key metrics. From a seller's perspective, this idea of when the sale is booked and the sale is, is ultimately, and the deposit has been recognized and the contract has been inked, that relationship isn't done yet, right? That's the beginning of the relationship. That's when we get the initial commitment. But ultimately that salesperson and part of their compensation has to be directly attached to usage. So just because you signed a million dollar contract with the customer, maybe we give you half credit or 2/3 credit for that initial commitment, but the rest of it is associated with making sure that that commitment is ultimately recognized as revenue, as the customer is getting value out of the platform. A lot of sellers do not like that. A lot of sellers, especially who like to think of themselves as hunters, see this as a farming exercise. But the reality is, is that's the relationship with the customer and that's ultimately the business model. And it's not just about aligning your own internal resources. It's about aligning your interests with your customers and making sure that your relationship with your customer is a partnership. So they have to recognize that when they make requests to the platform, we want to make sure that the requests that they're making are ultimately going to drive maximum value for them as a customer and ultimately maximum usage out of the platform. Because what's true in our world of software development and software delivery is that the more you build, the more value you get. And so we always want to make sure that when customers make requests that they recognize that the requests that they make are trading off against other things that might be incredibly important to them and others that drive usage. And so you always want to keep that relationship and that conversation open and amongst between both the platform as well as the customer. And so usage based businesses aren't really SaaS based businesses. In a usage based business, customer value and usage actually directly equates to, to company value and revenue. And that Direct relationship is so incredibly important and really reconstitutes the relationship between you as a platform to your customers. They become no longer just consumers, they're really your partners in growth. And so that's, that's the end of it. I mean, again, just sort of my takeaways. You always want to make sure you're matching your value with how you price and package to make sure you're treating your partner, you're treating your customers as partners in their growth as well as yours. And then ultimately everyone has to be responsible for the success of the customer. The, the inking of the contract is only the beginning of that relationship and everyone needs to be all bought in to make sure that that's successful. Great. So thank you everyone. Let me just jump down to the Q and A here and see what kind of questions I can answer for folks. All right, so the first question that someone asked anonymous attendee, does freemium generate larger customers or is it mainly spreading the word? Or do enterprises bypass free for the most part? So in our particular case, what we find is that free is the tool that drives trial for all developers, regardless of the type of customer that they're in. So they might be part of a Fortune 100 company or they might be part of a three person startup, but ultimately the needs of the developer are developers don't want to integrate tools that they don't have the ability to kick the tires of and they want to use the product. So it is pretty universal across all different product types. Second question is anonymous attendee asked, is freemium table stakes for developers now? Atlassian and New Relic have added free editions recently. I think that is true. Freebie in some version has to be available for customers and the two ways that you can get to it is if you are a cloud based service, you can get to it through a free plan or a free offering. If you're an installed product or a downloaded product, you can get there through open source distribution. But in some way, shape or form you have to get the product into the hands of the user so that the user can get familiar with and comfortable with how the product works. We actually internally we changed our free plan just a few months ago because what we found is that when we started with freemium, it was sufficient just to give customers just a little taste, just a little understanding of how the platform worked. And if they needed more, if they needed more advanced compute types, if they needed more advanced services, we would just expect them to pay at that point. And what we realized is that that line had shifted that now customers not only need a taste, they actually need access into the whole suite of services so that they really understand what they're buying when they decide to put their money on the table to actually buy the service. So I think you're finding that free is ultimately going to be the responsibility of everyone and you're going to find it across all, all platforms and vendors. Next, anonymous attendee, how do you determine the right choke for a free to paid upsell? Can data tell you or is it more rough experimentation? I would say ultimately data tells you, but it is all about experimentation. I think what we have found specifically in our business is that there are key moments in how often a customer builds when we know that they've gone from sort of open source and sort of side project and trial and they're moving into more commercial development. When they move into commercial development is when we sort of change that relationship and it becomes more of a commercial relationship. And so we're always keeping an eye on the data and we're always keeping an eye on really understanding when that transition point is. And like I said, we actually move the freemium line just in the last couple of months because we realized that what used to be that transition point between kind of a trial project and commercial development had shifted. And so we stuffed more value into our free plan because we realized that at the end of the day, free is an understood and capped cost. And once we can build that relationship with the customer, we get integrated into their project. It provides a very nice kind of ramp to growth. All right. And I think for any platform, you really need to know, just look at your data. And I think there's usually these kind of key moments and key transition points in that customer relationship. All right, next, what have you learned about paying the sales team on usage based deals? Excellent question. And this has evolved over time. And we're always talking to pretty much everyone else that's in the usage based world to figure out how they do it as well. Nobody really has the Rosetta stone for this, and we're all kind of evolving and improving this as we go. But I would say best practice today is you pay the salesperson partially for generating that initial contract and that initial commitment, and then you pay part of their commission based on the actual consumption of that commitment over time. You don't want to pay 100% upfront on just the commitment because you run into situations where salespeople might oversell the customer and the customer ends up with way too big of a deposit and way Too big of a balance and no way to actually sort of spend that down over time. On the, on the flip side, you may end up in situations where customers grow much faster than you would expect. And you want to make sure that the salesperson is getting credit not just for generating a commitment that might be too small, but ultimately is then getting paid out on usage that is eclipsing that commitment over time. So you really want to balance those two out. We've heard 70, 30, commitment versus usage, 60, 40, commitment versus usage, 50, 50. We're kind of somewhere in that range and we're always playing with it. But I think balancing the two is really, really important. All right, next, anonymous attendee. How has CICD changed over the past three to five years? What does this mean for founders and others trying to develop product more quickly? That is a great question. I think the thing to think about, how has CICD changed is when we started CircleCI 10 years ago, this notion of continuous delivery, this notion of automating the software delivery toolchain and the software delivery process was really pretty fringy. It was the big fangs, it was the Facebooks, Amazons, Netflixes, Googles and Etsy's of the world that had very sophisticated custom tooling stacks to support it. And then small, very technically sophisticated startups that were trying to emulate that practice but didn't want to invest the time and effort into having you manage a platform that wasn't core to their business. Now I think you're realizing that that idea of continuous delivery, this idea of as fast as you can build something and ideate something, you should put it into the hands of the customer, is no longer something that's on the edge. I think it's the aspirational North Star for every business that's out there. And what delivery means has changed. What used to happen 10 years ago is you might release once a week. Now when we look across our most sophisticated customers, they're releasing potentially hundreds of times a day. And as fast as you're able to conceptualize a potential fix or a potential opportunity, you get it into the hands of the user. And I think that that is going to be not just the aspirational North Star, but it's going to be the expectation for how fast innovation gets into the hands of the customer. And I would encourage anybody who's a startup to automate as quickly as possible because it frees up your developers to do kind of more important, higher value tasks that help you focus on what's really core and important for your business. Your customers don't actually care what CI CD tool you use. They care that you're a great payment service, you're a great crypto service, or you're a great game and outsource that. Put that on other third party vendors like us, or put security restraints and controls on security vendors. There's great platforms out there that can provide all that. You don't have to worry about building yourself Next Anonymous attendee since CircleCI builds in the open, what do you get feedback from your top 10 customers if what they're looking for is not on your roadmap or is coming too late for them? Excellent question. So if we are building in the open and we have a public roadmap and we share it with our customers and we share it with the market, what happens if what we have on our roadmap actually doesn't meet some of our customers requirements? That is the whole point of publishing. It is that oftentimes we don't actually know what is the most important requirement from a customer until they actually see that it's not being built. And then all of a sudden you realize that that kind of thing that was on page 20 of the RFP is actually number one and is the thing that actually has to get built. And so we are constantly talking to customers and trying to incorporate their feedback and adjusting the roadmap as we go. So our roadmap isn't really designed around sort of like quarterly slices. It's not like we deliver this in Q1 and this in Q2 and this in Q3. There's lots of folks that do that and there's nothing wrong with that. But that's actually not what we what the way we structure our work, the way we structure it is we say we're building this today, we're going to build this next. And then these are all opportunities and features that we could build in the future. Where that customer feedback becomes so critical and so important is that sometimes stuff that's in the future category needs to be in next. And some of the times the things that are in next can be pushed out into the future and you use that feedback to basically kind of move between those two columns. It's a great conversation starter and a great kind of trust builder between the platform and the customer. All right, another anonymous attendee in your model. Builders, sellers and marketing all have to work together to own the customer relationship. One sometimes when three people own one thing, no one owns it. What KPIs are sales, marketing and product responsible for so that the ball doesn't bounce in the middle when recruiting. Is this probably a different environment than what most candidates are used to? How do you train them to shift their mindset, I. E. If a salesperson is resistant and wants to walk away from the customer relationship after it's closed? Both great questions. The first is how do you get people aligned around specific, specific goals and KPIs versus everybody sharing one. There's that is a constantly evolving thing that we're managing. And the way that we've chosen to manage it today is to divide the notion between generating demand, which is a marketing concern of generating demand in the marketplace, and generating demand with customers. Our sales team is ultimately responsible for generating commitments and getting those initial commitments signed. And they work with marketing to basically process and transform that demand into commitments. The sales team, as I mentioned before, the majority of their revenue or the majority of their commissions are tied to that initial commitment. But a part of their, a part of their commission is tied to usage, that usage. KPI ultimately sits in our success group, in our success team. And so the success team owns the usage number that then they kind of share alongside with the sales group. And so each group has their own lane, each group has their own responsibilities, but ultimately they share it across because at the end of the day, the way that we think about it internally is that the more usage, the more pipelines that are generated on our platform, the more value that we generate for, for a customer. And ultimately that's going to transform into revenue at some point. But the idea of trying to get people into their various lanes is really important. And then they share that kind of top line usage target alongside with product. From a recruiting perspective, you do run into all sorts of situations where as I mentioned, you have certain types of folks and certain types of sellers that love to be hunters. They love to be the ones that go out and create that initial relationship with the customer, but aren't necessarily as sort of interested in maintaining that relationship with the customer, making sure that that customer is successful, successfully using the product and successfully using the platform. Two strategies that we've seen, we've tried them both. One is you take the folks that are really, really interested in new and interested in securing new relationships and new logos. You give them a specific new target that basically you need to go out and generate X number of commitments with new customers and then you own the usage target for that first year. Sometimes that works, sometimes it doesn't, but we have tried that with some level of success. And then there's other times when you have salespeople that are truly just want to go out and hunt and don't want to do this kind of ongoing customer relationship management driving usage. Those aren't oftentimes the best recruits or the best hires because I think we have and the majority of our revenue comes from our existing customers, not from new logos. And so we have a distinct need and desire to make sure that our existing customers are successful and we're generating those partnerships with them and they may have a different sort of career goal and that's okay. You know, there are lots of opportunities out there for great salespeople who are really interested in just hunting and not doing that kind of customer management piece. And you can suss that out in the sales or in the recruiting process. Next, how does marketing change in a usage based model? Does the marketing start further down the funnel after the customer has already committed to get them to use the product? Another great question. So marketing actually ends up in our situation ends up being actually twofold. So we have one part of marketing that's focused on making sure that we can generate demand. So our whole demand gen team is really focused on being able to get new users, new logos onto the CircleCI platform. And then ultimately the product team and our growth teams are responsible for making sure that they get onto the platform, get their projects up and running, and that they use the platform. And that demand gen team is allocated and really focused on getting those new users new logos onto the platform in conjunction with sales development. On the flip side, field marketing and customer success work hand in glove to help drive additional adoption in existing customers. So in many cases, if it's a customer that we already have a relationship with, we already know who the key constituents and key champions and users are. That's really more of a customer success relationship where it's trying to understand what projects they want to build, how do we get them on, etc. But when you're dealing with big Fortune 100 or big Fortune 500 companies, there may be a whole collection of opportunities that are in the same corporation, they share the same, the same business cards, but are entirely different opportunities. And in those cases, we use both field marketing as well as sales development, working in conjunction with our sales team to find those new opportunities and to basically tuck them in as we go. But we're always evolving that approach as well. But we have found that basically marketing kind of marketing to new is slightly different than marketing to existing in a bit of a different motion. Next, an anonymous attendee. To what extent do you think the current SaaS product landscape embraces or does not embrace a usage based model. And why do you think this is so? I think there are more and more platforms out there that are adopting usage based models. Models. Should they all be adopting usage based models? I'm dubious. I think like for example, there's investors out there that are saying every, every platform is going to be a usage based platform within five years. I don't actually think that's good or I don't actually think that's true. I think for usage based models what you need to have is an alignment between how the customer generates value with the pricing model and in our particular case, usage base makes sense because in situations where you have very sort of spiky demands, making sure that people have the opportunity to sort of only pay for what they use is a really important sort of customer value. The second is when customers can't really determine what they want to buy ahead of time. A usage based model provides the buyers and the users that flexibility to be able to support it and not necessarily have to forecast ahead of time. I mean, in software development the really tricky part is you don't really know what you need to use until you need to use it. And if you had to forecast ahead of time, you don't have it at your disposal. And so usage based models make a ton of sense there. But I think the really important thing inside of usage based pricing is that a customer should get incremental value out of each incremental use of the product. And with each new use of the product, they get increasing value. If that's true, then usage based models make a ton of sense. So like an example would be. So in our situation for Circleci, every time a customer builds software on our platform, they're getting key validation and key management sort of feedback and understanding as they're delivering software out to the customer. And each build is uniquely valuable. But also the more often you build software, the more resilient and better your software gets over time. That's basically what all the research shows. And it's true based on our experience as well. But there's other businesses out there where that's not entirely true. Like I think about things like messaging. Messaging. The value is in just getting a user on the platform, right? And Whether I send one person, one Slack a month or 1,000 slacks a month, the fact that we're on one shared platform is really where most of the value is. There's not a lot of incremental value in every new message that's generated. Those kinds of businesses, I don't think lend themselves all that well to usage based models, even though I think folks are trying to jam them into usage based models. I would think long and hard about what and how the usage, how the user perceives the value they get from your platform and how that accrues over time before you go into a usage based business. Because usage base is hard. Usage based is very complicated to track, it's very complicated to manage, it's very complicated to forecast. You know, the biggest or one of the biggest usage based models out there is Snowflake. Snowflake had an absolute banger quarter this last quarter growing at over 100% at a $1.4 billion run rate. But when they forecasted forward for the next year, they're like, well, we think it's going to be somewhere around 65% and it's not because they think growth is going to slow down, is they have no visibility. You don't know what your revenue is going to be until the customer actually uses the platform. And that is something that I think we as platforms have to get prepared and really understand sort of how to, how to forecast that out. But I think the market also has to understand that there's this fundamental transition and fundamental trans transformation that's happening in the market as a whole. All right, next in the bottoms up model, how have you determined the threshold with which to engage reps, to reach out to customers, to enter into a committed relationship? Put another way, how have you determined the right threshold of when you should engage the self service audience? Excellent question. Obviously people build software on our platform and so we have very good metrics on exactly what customers are trying to do on our platform and how often they're building on our platform. What we typically look for are increases in utilization. We look for sort of duration of utilization as well. When like we don't want to basically because again, we're working with software developers. Software developers typically don't want to talk to salespeople or go to market folks. They kind of want to be left to their own devices to be able to kind of understand and test the tool without people in their hair. So we will more often than not, when they come onto our free service, we'll leave them alone. We'll enable them and empower them. Give them, you know, tips and tricks and tools and onboarding tools and everything else so that they can get their projects up and running. But once they get their projects up and running, we want to let that marinate for a while because what we have found is that by letting it marinate, by allowing the developers to really understand how the product works, eventually it will get embedded into their projects and then you'll see this spike in usage. When you see the spike in usage is typically when it's a good time to have a conversation with them, just to really help them understand, like, what's the right commercial relationship? Like, what can we do to enable them, how can we make their builds more efficient? Because that's actually one of the biggest services that we can provide is we understand how software is built. Because we see it built millions of times a day. We can provide a lot of that insight to our customers of just like how and what best practice looks like at any one given time. So there's not one answer. You basically have to kind of monitor exactly what your customers are doing on the platform and then, and then just run tests, trying to engage them at different points of the funnel. Next, the movie analogy, building in the open. How would you defend the famous adage, if I had asked, people would have asked for faster horses. It's a great question. I think the way that you try and get around it is we, from a product perspective, what we try and encourage people to be is problem focused, not solution focused. When customers are asking for something new, when customers are asking for us to adjust something, the question to ask isn't, is that the right solution? The question to ask is, what is the problem that they're actually seeking to solve? And then really focusing on that, focusing on what the problem statement is so that we can then take that back, look at that particular problem versus all the other customer requests that come in and determine what the right potential solution would be. So the best, the best advice I can give is just be problem focused, don't be solution focused. And that is actually a very big problem in our space because as you can imagine working with technical audiences and working with developers, developers love to tell you what to build because that's what they do, they build things. But you know, you want to sort of not shortcut to those answers. What you really want to do is sit with them and understand, like, what's the actual problem that you're trying to solve and what are some potential ways that we could address that. And that's usually a great conversation starter and we use that as an opportunity to get closer to the customer. Next, how do you handle a situation when customers consume.