How to Successfully Bring AI Products to Market at Scale with GitHub’s CRO

Important in all of these sales conversations is relentlessly quantifying value. I've never seen from customers such a voracious appetite to measure ROI from a product. Right. They want to know from day one what's happening in the product and what benefits they're getting from it. And this is hard to do well, right? You need the data emitting from your products. You need the context from your customer to understand what they really care about, which, which of course is different for every single customer. And then you need to package that into a narrative that makes sense. Please welcome Elizabeth Pemmerell, CRO at GitHub. Oh, saster. This is my first time here. I'm thrilled to be here and have had conversations with a few of you. I hope I get to meet more of you. I'm Elizabeth Pemerell. I am the Chief revenue officer at GitHub. If GitHub is new to you, GitHub is where software development happens. We have 100 million users on the platform, 90% of the Fortune 100. It's where developers create and build and secure and scale. It is the home for software developers and GitHub is also my home. I've been at GitHub nearly a decade. I joined as an individual contributor. I was the first seller focused on our public sector markets in the United States. And since those early days with 200ish employees to today we have thousands to now over $2 billion in revenue. It's been an incredible journey. As CRO, just for context, my responsibilities include all of our customer experience teams so sales, our technical sales teams, post sales functions like supports customer success partners, expert services and our go to market with Microsoft since we were acquired in 2018. And we have been on a journey. We've been on a journey with Copilot. Copilot. It started really just with our interest to help developers be more productive, to be happier, to stay in their flow state longer. As we went through this journey we realized there's actually a product here. And so we brought the first iteration, Copilot for individuals to market. We then brought Copilot for business to market. That was early 2023 step ahead to today. We have 77,000 organizations using GitHub Copilot and almost 2 million users in the last fiscal year. Our fiscal year ends in June. Our AI products were 40% of our revenue growth during the period. Pretty remarkable. So we're talking I think pretty real scale here and we learned a heck of a lot along the way. I'm going to share some of those perspectives with you, and I'm hopeful they may be useful. In that video I mentioned being customer zero. This was a very important component of our journey. As I mentioned, we set out first and foremost to build a product we wanted to use internally. Copilot, at its very nature is code completion. So our developers start typing code and the copilot then offers the rest of that coding suggestion based on their prompt. And as I mentioned, first and foremost, our individual developers were using it, they were loving it, and we thought, there's a skew here. This is its own product. We brought that to market and as we used it more, we realized we're going to need user administration features, better onboarding capabilities, SSO integration and the like. And so that wrapper became what is now the SKU we call Copilot for business. And we continued doggedly poking at this problem of where else can Copilot help developers be more creative and more satisfied? And so we integrated Copilot then across the command line, across our pull request features, across issues and documentation, and that became the basis for Copilot enterprise. So what I think we did well here was stay very focused on the problems we were trying to solve. And that discipline is incredibly important as we think about scaling AI tools because A of the number of use cases you could go tackle and B the ubiquity of A and I now solving all the problems. We stayed really focused on the problems we wanted to solve and we didn't get distracted. And that distraction is also ubiquitous, right? We have new products coming to market, we have new companies being born and funded, we have new models being deployed constantly. We've stayed focused on what we want to solve and yay, good, great, good job. Us, we get this product out in the market, but with developer tools in particular, you ship something to the world and you immediately hear about all the things you didn't get right. There are so many software development languages and IDEs and technology stacks and flows that the feedback came pouring in, right? And with the volume of feedback and how dynamic this space is, AI in general, we knew we needed a much stronger feedback listening mechanism in place between our customers, between our revenue team, between products and engineering. And that listening mechanism is somewhere. We had to invest a lot really quickly to make sure we could capitalize on the opportunity in front of us. And so we set out to re architect this whole thing. We focused on three major components of our listening system. The first was making sure the process was as seamless and friction free as possible. We then wanted the door to be as wide as possible to get that feedback. And then we've had to challenge ourselves to be very disciplined with that feedback in prioritization and communication. And so to touch on the first that the process component. Historically the burden of customer feedback fell on our solutions engineers and our customer success architects, bless them. We would send them this incredibly long issue template and say we're going to need a brief 47 fields here on what the customer told you. And sure the feedback we got was great, but I think we were capturing about 30% of it right because it was those dear hearted teammates who took the time to do it. We blew that apart and we created a simple chat slash command slash customer feedback. Now it is seamless and in their natural flow to add feedback right where they're working day in, day out, which lands in GitHub issues, then triaged into a central project board and now weekly. We can tag it, we can aggregate it, we can add value to it and we can make decisions based on what we're hearing. So we started slow. We got this right with those dear SES and CSAs who are on the journey with us and then we started to look at that front door. Like I mentioned, where else do we have feedback opportunities across the business? Our support team, naturally they are on the front lines with our customers every day and had lists upon lists of things that they wanted to see product prioritize. We started to incorporate that into the process. We then looked at our customer advisory board feedback, our executive briefings, et cetera. Couple of benefits of our investment here, A getting decisions made and those decisions communicated back to our customers. B we now are seeing the percentage of those issues that we work and resolve, quarter over quarter steadily ticking up so we know it's working and see for the leaders in the crowd, our semester planning process, which are these sprints where we decide what we're going to work on. Used to be this weeks long thing of passing documents around, lots of like really strongly held opinion loosely backed by data. Now we pull up our project list and at our last off site it took about two hours to go through the list and say what are we doing, what are we not doing and why? And so that provides tremendous velocity and also gives us a lot of time back to focus on more important things. And so when we think about this in action for Copilot for Business, an interesting scenario arose where we put this into practice. When we took Copilot for business to market, we knew of course this was going to be valuable for GitHub enterprise users. So folks who actually have their code stored on GitHub, and from a go to market perspective, it was pretty straightforward. This is a cross sell. We're going to take this SKU to our existing customers. We have commercials with them already, off we go. And so we're out running in the market talking to customers. And what surprised us a bit was the demand from prospects who said, we want Copilot. We said, you're not using GitHub. And they said, but we still want Copilot, like, oh, okay, tell us more. And so we started to aggregate that feedback and thought, huh, this could be an interesting new direction. And so through that feedback process we're able to take in different TAM analysis and customer segments and the like and we made the decision to move forward and prioritize a new direction for the products. And for me, this process also created a bit of a mental model on how we're thinking about AI products that may be applicable in other cases. First and foremost, as we tinkered with what became Copilot, we had to make a decision. Is this code completion functionality a capability that's part of our native platform feature? Okay, is it something now that we want to independently skew or make part of the platform SKU that already exists? Okay, we want to natively we want to take that as its own sku. And now that it's its own sku, is it a SKU designed just for users of our for product or platform? Or is this a SKU that's going to open up new and adjacent markets? Okay, that's the decision we made. And then how do we price all of that so that the pricing of the SKUs you end up with at the bottom of that decision tree makes sense relative to your base product pricing? So as I said, the solution was we ended up shipping Copilot for business for non GitHub enterprise users. It's been remarkably successful commerc. It's also opened up relationships to net new customers for us that we certainly hope we can move to the GitHub platform over time. Another place we saw this in practice, this feedback loop working very well was related to adoption metrics. Customers crave metrics about how their teams are using all AI products, GitHub Copilot included. And once we shipped our first version of our API again flooding feedback that we didn't include quite the right things. They needed a little more here, wanted us to add this and that. We had to have a feedback loop in place in order to prioritize that and make sure we were going to ship the next best things. The value here was also in us deciding what we weren't going to ship. And that's where we've really leaned into partners to fill the gaps in some of our product capabilities and make room for them in this copilot ecosystem through things like copilot extensions. So we've learned a lot from our customers as we've gone on this copilot world tour. What was interesting to me as well was how much teaching we had to do and think. Like 18, 20 months ago, our customers were on the very, very early days of their AI journeys. And so they wanted to know about privacy and security and legal. And we spent a lot of time educating them, getting back to basics, clearing up misperceptions, pushing back on misunderstandings. And I would say a learning for me was that this was the most uncomfortable. I think I have been consistently in sales cycles because we had to really lean back on customers and change their thinking about what it meant to adopt an AI product. So don't underestimate the time that's gonna, that's gonna require and the, the discomfort it's going to cause. But I'd say we came out stronger on the back end of that because we've only solidified our trusted relationship status with these customers. Okay, so now you have a good strong product in market, you're listening to your customers, your customer, zero. Not only did we have to learn and listen and adjust on the product front, we had to do this on the go to market front as well. And part of this was because there was absolutely no playbook we could pull off the shelf 18 months ago and say this is absolutely best way to take an AI product to market. Didn't exist. There was not sales leaders or sales teams who had done this before. So we build the playbook as we go. An assumption we had, for example, out the gate was that junior developers are going to love copilot. Companies are going to want it to give it to all of their early in profession talent. It will make them more productive more quickly. Lo and behold, we start talking to customers. We realized actually there was far more excitement about the productivity it was giving back to senior developers who were able to craft more sophisticated prompting and focus on even tougher problems for the company when they weren't caught up in rote administrative tasks. So we had to go back to our PMM playbook and pull it apart a little bit and start over with what we thought was going to be that initial value proposition. We were able to do this well because a core ethos in our company is being iterative and being a little bit uncomfortable. We have two core company principles of seven ship to learn and embrace a growth mindset. And these aren't just things we talk about in all hands and put on slides. We actually have emojis in chat. And when we see this behavior, we celebrate it. We talk about this actively in our reflection conversations when we're considering performance and impact. The other concept that we've really embraced in revenue is hiring for low ego and high eq. And I've heard this concept a couple of times today from other speakers. We want to look for people who have been through something, who have stuck with it when the quarter or the year has been lean. They've gone through a messy acquisition, they came out the other side, they have a story to tell. And as a result of this, we have a team who really leans into change and is excited to embrace the things that are messy. The other way we embody this is that we run six month compensation plans for those unvariable plans. So, so six month plans, every six months you get a new exhibit. So we're in constant planning mode. This takes a lot of work from the sales leadership and our finance team. But a couple of things. It means we can really quickly react to what's happening in the market and we can make adjustments as we go and we can tinker, we can try things for six months and see what works. Now, six month comp plans, a lot of reps are going to say I don't want you to change my plan after six months. Sure. There are times when the plan benefits the company. More often than not though, we found that the six month planning process benefits our hubbers because we have something unexpected happen in the products. We decide to make a change to segmentation or alignment or we have macroeconomic situation that we didn't expect. And so it sends a pretty strong signal to candidates who are interested to join us when we let them know about this six month plan. In terms of other pivots or learnings in the go to market journey, we've had to adjust to new sales stages and to new buyer Personas. So I mentioned before the proliferation of conversations related to legal and privacy and models that took much longer in all of these sales stages than we had ever experienced before. So we had to take a new approach. Right? Our Solutions engineers, our CSAs, our selling teams, our support teams were answering so many questions that they could not keep up with the demands. And so our legal team to their credit, took the matter into their own hands and went on a roadshow globally. They hosted roundtables in major cities around the world where they invited the GCs of our customers and said, let's have a frank conversation about AI, what it isn't, what you shouldn't be concerned about and what you probably do need to be concerned about and thinking about. They paved the road out ahead of us and became really an extension of our sales organization. So this was great two years ago, but of course didn't scale. So we had to translate those materials to self serve and we did that via this copilot trust center. We put all of that material in a place that can be accessed by anyone in the company who might need it, whether it's a one pager or simple videos explaining how the product works. In addition to these new kind of major steps in our sales cycle, we're seeing now as our customers get more up to speed with AI and more comfortable, there's some new buyer Personas emerging. We've had some customers name AI transformation officers or chief AI officers that will be one of our buyer Personas in the years to come. And I think it will be very interesting for us to think about how that role intersects with whatever our kind of core traditional buyer Persona was, where the roles, responsibilities and budget lie and what's important. So that's something I'm definitely keeping an eye on. Important in all of these sales conversations is relentlessly quantifying value. I've never seen from customers such a voracious appetite to measure ROI from a product, right? They want to know from day one what's happening in the products and what benefits they're getting from it. And this is hard to do well, right? You need the data emitting from your products, you need the context from your customer to understand what they really care about, which of course is different for every single customer. And then you need to package that into a narrative that makes sense. And so A learning for us in this process was A making sure we were getting that right, data emission, and then B, making sure we were investing in things like value engineering, external references, case studies, internal metrics from our use cases or from Microsoft's use cases. So don't underestimate the investments you need to make here in order to be really compelling to your customers. One more learning. When we first were bringing copilot to business to our enterprise customers, there was also voracious appetite for pilots. Everybody wants to tinker, they want to try for their use case for their team for 10 developers, that's very exciting. It is enormously taxing on your organization, especially when you're small. We very quickly had far more demand than we could handle and especially from companies the size of Accenture, they can eat you alive. With a poc, it could take a year. And so we had to get pretty opinionated about what we felt was a POC worth supporting and how we were going to measure the impact. And so with Accenture they started with 20 developers internally and they said this is pretty good, like we want to do something a little more formal. They set up a trial with 450 developers using Copilot actively for a period of time and tested that against a control group of another 200 developers. And at the end, not only yay, did they buy Copilot and roll it out to thousands of people, but they produced this study which is also very great, a nice selling tool to have on hand. But the bigger lesson there is we have to get pretty disciplined about saying no or pushing customers to a package sort of POV perspective. The same is true post sale for adoption. Customers say, well we'll let the developers use it who want to, let's get everybody going here. So we had to be really prescriptive about that adoption journey. And our CS team did a fantastic job creating the Copilot adoption framework and open sourcing it so every customer has access to it. Okay, so now we have our product in markets, we're getting feedback, we're learning, we're iterating on our go to market strategy. If you're lucky, this massive business you're building is now showing signs of strain in certain areas because of the incredible growth and that certainly happened to us. The next horizon is scaling. The single greatest asset you have are the hours of the experts on your teams. Solutions, engineering, sales, support, cs. We had to figure out how to make our sellers as productive as possible in a time when they couldn't possibly keep up with the demand. That meant putting as much as possible into self service channels. It melted. It meant enabling a partner network to help us recording things and then driving to one to many. So what used to be a bespoke let's have a call and do a demo situation quickly turned to you are welcome to join the call with the rest of the Belgium next Friday at noon. And we were unapologetic about that because we simply couldn't keep up with the demand. The other thing we had to think about pretty fast were some of the infrastructure elements that needed a little investment on our side. In times of massive scale, it's a really good opportunity to get more opinionated about your pipeline because you got a lot of it coming in. Now's the time to get the baseline in place of how you're going to measure it, how you attribute it, what levers you have to pull around it. Because when times settle and now you're in run rate or maybe even lean months, you're going to want to know a lot more about where that pipeline came from so you can go recreate that moment. Second major thing we invested in in the most recent year were digital moments. We need our customers to be able to do everything themselves on demand, without human intervention. You can brute force these things in the beginning, right? Sure. The SDRs can complete that. Sure. Sales operations can just create a few quick salesforce cases. It doesn't work at scale. And we started to collapse a little bit under that process and administrative technical debt that we created over the years. And finally we're really investing right now in all things related to our seller productivity. So every little paper cut we possibly can, we're trying to reduce every click, every step that's extra or unnecessary, we're trying to eradicate it. And we are on a campaign this year to make our sellers massively more productive by reducing that paper cut. Now, when I talked about being overwhelmed by the scale, this was true also with internal processes and experiences like support. And so we turned to Copilot to help us actually with our support case queue. We experimented here a little bit and actually ended up implementing Copilot as the front line now across many of our support surfaces. Based on the success that team has realized, we estimate that we're deflecting about 60% of the tickets we would have expected to be created across our support services, which is incredible. That's savings of tons and tons of man hours and hires, which means now our support team can focus on more complex problems, higher order issues. And we also now have this wonderful treasure trove of data to help enable our new support hovers when they join us. A couple other experiments we're running internally. I haven't found any other examples yet that we're saying are winners. We're definitely tinkering in lots of places, but our COPILOT for support implementation has been a winner. And then partner strategy. I mentioned this briefly. We arrived at the point where we could no longer support the weight of our own go to market. That means both our sales motions and our post sales adoptions motions. We needed access to new channels in new geos and then we needed those partners to come behind us and make sure our customers were adopting the product and that their ARR was increasing year over year. And so we've made a lot of investments here in the most recent year. It's something to think about. It's a tremendous investment of time and energy from you and your leaders and your teammates. So you gotta wait until you have the capacity to do it. A couple of things we added in the most recent year that have helped a lot are more enablement sessions. Far more trainings are now available to our partners that we're running both synchronously and asynchronously. We needed to get our partners better access to our products. So now we have a much more robust beta program. Our customers get. Excuse me, our partners get bespoke roadmap sessions. We just did a partner kickoff this year for the first time. So we did our revenue kickoff, then we did our partner kickoff for the year and we released a certifications program. And this is really helping our partners self serve that knowledge they need and make sure that they're ready to talk to customers. There's been a lot going on, it's scaling really fast. There's been a lot of noise. We took a step back probably a year ago and said, gosh, how do we help make sure we're driving signal through all of this noise and change? And we doubled down on the fundamentals in a very specific way. We took all of our go to market motions and our conversations across the company and organize them through the lens of three revenue plays. And if you ask any hubber, not just a revenue hub or any hubber, our three revenue plays, they're going to say land secure accelerate. Land secure accelerate. Because the revenue plays not only became the basis for our enablement and PMM road playbooks, for example, they also became the core of our product roadmap. They became how finance reports on the business. They became how we run our mbrs and our qbrs, how we measure our pipeline. And it was very simple shift we made to name these plays and to create clarity around them. But I have been very surprised at how the business has reacted to that structure and how much clarity and transparency we've created from end to end in the business. Not rocket science, but a simple shift you can make to help make sure that your teams are all aligned. I want to just hit on these three points one more time before we break. I think we did a really good job keeping customers at the center of our journey, scaling an AI product to millions of users. And we did a really good job pivoting based on their feedback, because we created this very functional feedback loop between customer revenue, partner, engineering, product design. We then made sure we were very nimble in our go to market. We adjusted where we had to, I think, buckle up. That's not going to stop anytime soon. And to do that, we had to make sure we had that ethos in place and we had the right people in seats to make that a reality. And then we've been frantically investing in our scale engines to make sure we're ready to get to the next couple million users of Copilot. We sure hope. And all of this was wrapped in a little bit of luck at being the right products in the right market at the right time. And so I wish you all that same bit of luck on your journeys. Thank you very much.