What Engineering Really Needs: Developer Experience as a Business Priority | GLOW 24

It was certainly very, very fun to see the DevEx launch happening live this morning. I know a lot of big hard work happened to make that a reality. We're going to be excited to talk a little bit about that. Just a little bit.

I may be slightly biased because I know just how much work went into to that and it wasn't just like, oh, the team did a lot of work because the team did a ton of work and you could see all the other parts of the organization jumping in and helping. It was really cool to just watch all that happen and I got to be a part of it and that's what I'll talk about today. But yeah.

I'll go ahead and move it to our title slide in case anybody was in doubt about what we were talking about and why the DevEx launch was so exciting for both of us here today. What were your other favorite parts so far today, Randall?

I love the second keynote and hearing about just all of the adventures she's had and just that drive to see a problem and say, can I do something about that? What can I do? Because so many people just go, well, that's a problem, I'm just going to complain about it. Just so good to see and have that enthusiasm of there's got to be something. So do something. It is possible to drive good change in the world.

Yeah. Yeah. I love that she called her generation hotheaded in a good way. Like, go for it. Just give it a try. Yeah. Yeah. Yeah. I'm going to talk about some angles of that here today. So I think it might be about time to go ahead and get us kickstarted. I'll go ahead and introduce myself and say I'm Marilyn Cole. I'm an engineering manager here at Jellyfish. Thrilled to be able to talk to you about the first half of the DevEx journey and how you make a business case, think about your questions and your timing strategically, and how you roll out a survey. And then I'll hand it to Randall to talk about the second half of the DevEx journey.

Yeah, I'm Randall. I'm a staff software engineer here at Jellyfish, and I was tasked with looking at the results and saying, all right, what, what do we do? What do we do now that we know these, these things to, to make and drive real change?

Yeah, I'm so, so excited to be doing this with you today, Randall. I feel like perfect symbiosis going on here, like talking about it and also, you know, working together in the work too. So it's a good thing. So thanks everybody who's, who's here to join. We're super glad to ask— answer questions. So definitely feel free to drop those in the Q&A as we go along. So yeah, how do you decide to run a DevEx survey? If you click over into the poll right now, you should be able to see a mirror, some, some, some direction, one direction of where this, this is in the polls on the side, the right side. You can see that we want— we're curious to hear from y'all how many of you have actually run a DevEx survey already of any kind with your team, whether it's in Jellyfish, a Google form, using something else. And wow, there's, there's, we're like even running so far, 50% of people have run surveys in some form or fashion. 50% of people have not yet run any kind of survey with their team. And to them I say like, what are you waiting for? The point is, like, quick. It was analysis this morning, if you weren't there for the keynote, the DevEx product in beta is now available to every Jellyfish customer in the app. So check it out. If you're an admin, you can go ahead and start looking at questions and thinking about them, which is the— what we're going to talk about first. Like, how do you think about the timing and the questions when you're getting ready to run your survey? Um, for those people who haven't done one yet, I would say truly go ahead and do it. Like, the timing is now, um, because there's so much energy in the market for listening to your engineers to help think through what, what will help them be more productive. Like, when you have Jellyfish, to be able to see the quantitative and the qualitative side by side, it's, it's a perfect opportunity to use that soon. If you have run a survey before, I think the important thing to think about with the timing is what have you done since the last time you've run that survey? And do you have any other surveys going on at your company at this time? So for us, we have a big DEIB and engagement survey that runs about every 9 months. And so trying to line up like big survey pushes so they're not all at the same time, like at the end of the year or at the end of some quarter, makes a lot of sense so that you are getting as many people as you can to respond. But yeah, if you have run a survey before and you're not sure yet, like, have we made progress on some of these things that people noted were concerns, or we haven't talked enough about the progress that we've made. Like oftentimes teams see the results and do make progress, but don't necessarily publish all, publicize all the things that they've done. And so it's certainly worth talking about amongst your leaders and your teams, like what are the things that we've done since last time and how can we think about those and talk about them in a way that enables others to make things better and encourages people to be thinking about how to make engineering faster. And the other angle that you're talking about your DevEx work with is with other leaders throughout the company. And so when someone might be asking, like, we just want to move faster, like, what are these engineers, you know, what are they going to be talking about? What is their job satisfaction? Engineering is the most highly paid organization at the company. So like, why do we care about their satisfaction? Well, that's one reason. It is important to retain those engineers. But another reason is almost every tech company these days, almost every company has some kind of platform team to help their engineers do better work. And so the, the importance of validating what that platform team is focused on is critical to all of us. Like, even if you don't yet have a platform team, if you're that small that you don't yet necessarily have a platform team, you certainly have some engineers who are working on this on the side. And that's kind of antithetical to the jellyfish missions, like people working on things on the side that aren't getting visibility. And so when you can think through for these, like, you know, very highly compensated roles how to ensure their work is the most important work, you think through like this stat on DevOps, the state of DevOps, and how many people in that part of the world think their number one problem is understanding their internal users and how to make things better for them. So in the same way, throughout Jellyfish, we're focused on giving you the data to make better decisions instead of relying on gut and who makes the best, like, most charismatic case for, for a specific thing to work on. This is about looking at the full data for your entire engineering team to say, like, what are the things that are holding us back and what are the things that we think will help us move faster? And so with that, we think we want to think through, like, as we think through the timing, as we think through the things that we know our platform teams are working on, our infrastructure teams, our engineering efficiency-related teams are all thinking through, we also want to know, What questions, especially if you're on your second version of the survey at this point or your, your second round, then like, what are the things that we wanna focus on and make more clear for this strategic moment in our business? Sometimes, you will be in a place where the question that you asked 6 months ago, 9 months ago doesn't make sense for where you are now, or you wanna add an extra lens for how you consider it. So you want to be strategic and contextual in thinking through how you frame questions. So for one example, in our case, we had in our, one of our very early DevEx surveys over a year ago now, a question, I think we still have it in our survey around for us, and certainly it's available for others, a question around, am I context shifting a lot? Obviously this is a thing that like a lot of, all of us deal with in the modern era. Like we have a lot of things coming at us every single day.

right?

And you wanna try to protect your engineers from that to some degree. But if you're at a fast-moving startup, you're not gonna be able to completely eliminate context switching. And so in addition to now giving the ability in DevEx to allow engineers to actually sort which things they see as the most important to do something about, we also added a question for us that was more around Yes, you might be changing context a lot, but do you also have the information you need to make good prioritization decisions when you do have competing priorities? And so there's two angles on it. Like, are you getting the information you need? If you have two things coming at you, do you have the agency, the A in space, to actually like make decisions for yourself instead of being bottlenecked by having to check with somebody else? Like, is this more important or is this more important? You can actually move forward, say, I can't do that right now, this is the most important thing right now, and be able to communicate that. And so we want our engineers to have that agency and ability to do that for themselves. So yeah, go into your surveys with the, the research-backed questions that we have, some of them being things that will potentially yield surprises, but also be ready to talk through it with your engineering teams. And once you have those results, you can talk through it and maybe even kick off a few things you want to do with it. And with that, I'll hand it to you, Randall.

Yeah, so we had some results. We certainly had answers of some kind. We had said, okay, these things people really like, these things people are complaining about, there's some things in the middle. How are they, you know, are they— does everyone think they're in the middle? Do people hate them, like them? And digging through all of that was Honestly, a lot of fun. I got the privilege of being the person to really dig in and take a look at what could we do. We learned things, we talked about with the teams and what can we do about this? My job effectively was I got to make everyone happy, which isn't so much like, oh, now I need to make people please, but it's just like, hey, people complain about things and I get to go fix them and tell people, hey, the thing you complained about, it's better now. Which was a lot of fun. And so I took a look at the results and the first question was, are there surprises? Are there things that we were worried about that were like, oh no, it's like, are people context switching constantly? Is that a huge burden? No, it isn't. Okay, good. And you know, what are, what are we doing not so well that we can improve on? And the two big answers that stood out was, you know, People complained about testing, our testing infrastructure and our documentation. So the first thing I did looking in that was looking at all the open responses to see what people were specifically complaining about, because those are big topics and spending all this time fixing something that isn't a problem is a great way to waste my time and I want to avoid that. I'm finding out what the specific frustrations are. One thing that was really helpful here was being able to kind of see results by team. If enough people from a team submit answers, we're able to confidently anonymize those answers and being able to see Our platform teams really were like, our infrastructure really needs some help. And some other teams are like, actually, that's fine. So I was like, okay, I know who to talk to. Maybe not specific people because they don't provide that. But I wanted to go talk to people in general because I wanted to tell people, hey, these— you put some time and effort and thought into answering the survey. These results matter. I posted in our team, just all of our developer channel on Slack and got this enormous discussion. Lots of conversation. It was fantastic. All those people talking about, hey, here's some problems, things are really, you know, stumbling blocks or, you know, broken stairs I stub my toe on every day. It was great to hear all of that and then get specific answers. And so, you know, digging in and saying, okay, what are the pain points? You said you're dissatisfied with testing infrastructure. What is the problem? And the two big things that came out of that were we had flaky tests. They weren't too flaky, but there are still people who are hitting problems where it's like, oh, CI failed, the build failed, was it because of my changes? Then you sit down and investigate, no, it's that one dang test again. Then rerunning it and it's just people talked about even when they did write something that test failed, they just rerun it automatically because they assumed it was flaky and then it failed again, they go, I have to fix things. At the same time as well, just like some of the tooling choices we made, we made when the Jellyfish was first founded and no one had gone back and said, hey, is that the best choice to make? We're a very different company and it's a very different time and lots of cool stuff has come out in the meantime. And so, you know, writing tests just took a long time and, you know, it was an exercise in frustration about using outdated tools. And so I was like, oh, okay, we can, we can do things about that. And more importantly, like, this is impactful things. This is— engineers like to grumble. I know everyone in this room is either an engineer, has been an engineer, talked to engineers. We love complaining. We love just griping about everything. If I spent all my time trying to solve everyone's gripes, it would not work out well. But these two things were really deeply impactful to the business in terms of, hey, engineers are taking a lot longer to get changes out. It's impacting our cycle times and things like that. To say, oh, frustration, I got to rerun the tests again and again, or it takes so long, so much of my time is trying to wrangle these decisions we made when we were founded and maybe it's time to revisit them. Digging in, we said, obviously I spent about 2 weeks of my time, so let's timebox this. I'll spend 2 weeks really digging into all the technical things, just this list of specifics we'd gotten out of that. That's part of our answer here is all the complaints were technical. That was always made my life easier because I like working with that thing and digging in. And then this is Jellyfish, so we care a lot about business impact. We can have the, the, the, this brilliant testing infrastructure, the best in the world, and if it doesn't actually help us drive our mission forward of building great engineering leaders, then may not be the best use of our time. But in the same way, we're already measuring a lot of the things we care about, like how long it takes to deploy a change, how long it takes to move a ticket from created all the way to, to in, in production. We're measuring these things, say, hey, we put 2 weeks of time into this. We removed or fixed all the flaky tests. We had flaky test detection so you could automatically detect it, remove and fix the flaky tests, went through a whole bunch of Python dependency fun. Oh boy, I could tell you lots of stories about that. To make sure that we were using the latest and greatest when it came to all the testing tools and infrastructures. Then did a little roadshow because we could fix everything, but if people didn't realize it was better, then it's not that helpful. I did a little roadshow and actually it was so awesome to have like people realize, hey, we can change things, we can improve things, people chipping in and helping. I think we ended up with like 10 different people in some way helping out with this effort. Wasn't 10 people all working for 2 weeks, but someone chipping in and saying, oh hey, here's this thing I can do for you. Oh great. And just this enthusiasm and Personally, I don't think we have a number for this, but the Slack channel we have for testing, there's so much more activity in it now. So many more people participating, saying, ooh, check this out, Taylor, like here's some tool or way or technique, or I just built this thing that other people can plug into. Just people getting enthusiastic about, you know, hey, let's really make as solid a testing platform as we can. And so what are the numbers? What's the results? So the first thing is deploy times are 20% faster using an up-to-date just running all the tests and also not flaking and failing the test. Turns out things run a lot faster, so we're just able to get changes out to production faster. A 65% decrease in change failure rate, which sounds a little too good to be true, and I'll go into it in a second because it was already pretty low. We can actually have— because this is Jellyfish, let's look at the results. This is our Lifecycle Explorer. You can track, hey, how the lifecycle of a ticket from, hey, here's something we could do, all the way to we did do that. And at the bottom you can see the trend over time for how long it takes for a ticket to be deployed and that work put into, hey, let's make this faster. And it's just, I just love this graph of just boom, you just see it's so nice as an engineer to see the impact of your work like that. You can also see in our Ardora tooling that change failure rate You can see it's already pretty low. We had about 3% of changes have resulted in a bug, big or small. If it's a typo or anything like that, we consider it a failure. It's already pretty low and it went even lower. The changes went in in February. You see a lot of people really paying attention and then they've just consistently been low ever since. It's been really nice to see that. But of course, the ultimate result, we had this DevEx survey, we said, What are people frustrated about and complain and got these results. Here's the results from the previous one that we did. They're not good. That doesn't feel good. That scores out of 100. You can see all the strongly disagree and disagree about helpful feedback from tests and how useful all these tools we have are. No one strongly agreed to this. There's one or two people are like, yeah, okay. We just completed, I believe yesterday, wrapped up our most recent DevEx survey and for the world premiere of these results, so much better. Nearly doubled the score there. Is it perfect? No, but this feels so much better. People feel like they have new tools they can rely on to do their job better. I haven't actually looked into because it's so new. I haven't looked into the detailed responses or anything yet, but we just wanted to make sure to show you this. Like, hey, it's so nice to show this to the team and be like, hey, your opinion matters. When you fill out those surveys, which is oh so exciting to fill out a survey, that results in real meaningful change in the organization. I'm super excited. I feel like I'm bragging a little bit, but I'm proud of this work. I'm proud of the whole team putting this together.

Yeah, I will brag on your behalf so you don't have to do it as much for yourself, Randall. Like, this was such an incredible delight to find out that these results got improved so much because of what Randall said at the very beginning of his part around like timeboxing this to just 2 weeks. We looked at the deliverable in Jellyfish and really the vast bulk of the work happened in 1 week, at least as far as the visible work that you see in Jellyfish, as far as like the Git and the Jira signal, because there was probably at least a few days also of like talking to people, you know, that's the important part to kick it off. But the actual dev work was like almost all in one week and made this big of a difference with how people perceive their ability to know that red means red and green means green when they're running their tests. Like, it is just so great. Like, when Randall first talked about this, I was like a little bit skeptical. Like, this gets back to like making, making the case for doing this work, and I was like, are are you sure like a week or two can make that much meaningful difference? And like, I think there was definitely like a way to point to like specific small things that could be done that would make a difference. And then you get that compounding factor of other people being excited, other people seeing that you're making changes, like, oh, I could do something small to add on. It's just really cool.

Yeah, it's super cool. Just, just getting the ball rolling and having that momentum and watching people like our team for deliverables where it said, oh, I built this tool for specifically testing deliverable items and now the whole team is using it. It's just so cool to watch people just build and keep running with it. I can't claim credit for all of the change, but there's been just a lot of enthusiasm.

Yeah.

That's awesome.

With that, we're in the Q&A portion of our chat. If you have any questions about our DevEx journey, journey, things we've heard from customers as we've onboarded quite a few alpha and beta customers in this last 6 months or more. Um, we'd be glad to chat a little bit more. Yeah, thanks, Will, for the nudge in the chat. And anyone else who has any burning questions, um, I, I have a question. I can come up with a question for you, Randall.

What?

Have you looked at the overall round of survey much yet to see across the board what's the next thing you're excited to work on? Is there something burning from the latest round of DevEx surveys that you want to like?

I don't know. Honestly, I think documentation, there were certainly complaints about documentation and that's a harder, thing. We did work on some things. I, I don't think the scores there improved nearly as much, uh, so I think that's a good thing to just be able to find, like, hey, if someone went to the effort of writing things down and saying, hey, this is how this works, we should make sure that it's findable and that you can. And so that's, that's something, uh, we have some, some secret projects with IT, uh, so, um, yeah, I'm excited with that. Oh, we got a question. I think that one's for you, Marilyn. Sure.

Yeah, thanks for the question, Steve. He asks, how often do you recommend teams take surveys? And I've seen a wide range of suggestions. I have certainly heard of teams that do shorter surveys much more often than we do. We, we generally have been running ours approximately every 6 to 9 months or something along those lines. But I've seen other places that, that make their surveys a little bit shorter and try to do it almost every month. I think it depends on your business. Like, same, just as we were saying around the questions and trying to pick the things that matter most to you and what timing you have around validating your systems and infrastructure roadmaps. How do you think through, like, what— like, if you're actively putting a lot of time into engineering effectiveness, then probably more frequent surveys will be better. If you're, you know, wanting to have a little bit more time for things to settle in, like, like we did, then I think you can, you can take maybe every 6 months or something like that as a standard. Mike has a good question too. Do you incentivize these surveys to boost engagement, or does that potentially skew results? So the main incentivization that we used was the fact that each team has to have at least 3 responses, and that's configurable to be able to share results by that slice of your team. So I work with one team that only has 3 engineers on it. And so it's like, if we don't all 3 answer these questions, we're not going to be able to slice it by this team. And that's okay if that happens. That's okay. But like, I think everybody wants to be able to point out like what our team's specifically wanted or desired as far as the things that are most important to them. And so we incentivize to be a little bit of healthy competition. This team already has all of their results in. Does anybody got— your team probably wants to have their voice heard too, right? But we don't incentivize individuals because of the fact that it's anonymized. It's only revealed in terms of slices that are more than a significant confidentiality chunk.

We have some good questions in the Q&A tab. There's one, you know, who should own developer experience within an organization? I think that's a really good question. I mean, because in this case it was just, I am a staff engineer who is available and able to dive in on this. For us, it's, I mean, this is a hard question in any organization and figuring out, you know, if you're a startup with 5 people, if you're like us, we're a growth stage, we're trying to, we've got some of the works, you like it. We're like, how do we make this even better every day? And you have only so many people in your organization and they, what are they spending their time on? And for us it really is this sort of, we do have a bit of a separation between, it's like, okay, we've got teams working on the product and building cool things and team working on the infrastructure and trying to make sure that the teams working on the product can be as fast and efficient as they can. And so it's those, those sort of info side teams, the ones making sure that like we have enough database space and all the data ingestion and number crunching is working and running, generally are also the ones who kind of own developer experience. And partially just because they're the ones who interact with it the most, like, hey, I need a database to test this thing. They, they're the experts.

Yeah, absolutely. Yeah, I think there's, there's a lot of different pieces around like who owns what. And certainly you don't want to go into the DevEx survey with through our entire team, framing it as like, this is how leadership is gonna decide what we're gonna focus on. You definitely should be like fully owned together as like some of these things individuals can definitely own and make better and teams can own and make better. And so like, how do you talk through that with everyone? Yeah, it, it, there, there's certainly some things that, that will require leadership conversations. And so I think that's really important too. To make sure that your entire engineering leadership team is talking through the things around, are we sure everyone is getting clear direction and why?

There's one on how do you come up with good survey questions? Product plug, use the ones that we provided in our DevEx product. We have some solid industry standard groupings for all of this. You can put in your own custom ones, absolutely. But I just want to note that, hey, we have survey templates that you can just use, and I believe you can get industry benchmarking on that. I hope I'm not promising something.

Yeah, all the questions that are available in the DevEx tool are research-backed. There is a set of recommended ones, and then in every section, as Sam was showing earlier in the keynote, in every section, if there's a particular section that matters most to you, there are extra questions there that you can use to to dive deeper on that particular area. And then that gives you a ton of good fodder to think through. Is there something that's really specific to us that's related to some of these? Can we think through it?

Yeah, I remember when Glenn was just starting off on like, how do we build a really good bank of questions? And he's just like, just anyone who research papers on surveys. And I'm like, Glenn, that's a huge topic. But he did. He told me and he just had like piles. I'm imagining, I don't know, like just piles of research papers around him running all that. They're probably digital, but he did a ton of reading and a ton of just like talking to people in the industry, talking to people in research about, well, how do you make a good and useful survey?

And I think we're gonna get cut off in just a second. Thank you so much everyone for joining. This was great.