Is Measuring Speed a Useful Metric for Design Systems?

Recording in progress. Okay. So I guess we are here because I upset Dan on Twitter. So I was tweeting my kind of distaste for speed as a metric for design systems, because generally— well, I'm not going to go into detail in the introduction, but generally I see it kind of misused and misinterpreted and not really kind of applied in the right way. And as Nathan quickly mentioned, like, very difficult to measure. Dan very politely ducked in and said, do you mind hearing an opposing view? Which you then shared, which you'll talk about a bit more here, I guess. And here we are to talk about it a bit more. Would you like to Start, Dan, maybe.

Yeah, I'll take it. I'll take it from there. So my first inclination was like, like, I'm triggered, so I'm gonna, I'm gonna say my opinion. And then I was like, wait, Amy didn't ask, didn't ask me for my opinion. And I was like, I wonder, you know. So I like paused for a beat and then figured like, uh, what do I have to say about this? And I think my first, my gut reaction to it, like, as I was thinking through it, I was like are we throwing the baby out with the bathwater? Which I am generally like hyper-tuned to. Like, I tend to look for that a lot where it's like, well, wait a minute, before we throw something out, are we throwing it out wholly? And I think like, I think design systems are supposed to help you be faster. And I think Amy, you like later on in that thread, you gave a good qualification. Like, are we separating speed of development and design and like process for the team with speed to market. And I think that's a good distinction to make, that like, you know, if we're conflating the two, we might— that might be a dangerous thing. So I think it's worthwhile. Even on that though, I'm like, I think speed to market is a good thing. I think that, that design systems are supposed to help businesses do business stuff, you know, otherwise they're just a design thing or an engineering thing or a product thing. And I'm like, I don't think that they are. I think they're a combination of those. So I'm like, I do think that design systems should help speed to market as well. But totally agree that that could be misused and misapplied, like you were saying in the intro. So I won't go too far here. All to say, like, that's what kind of provoked me to go like, what do I believe about this? And how does that match up to what other folks believe about this who do this work a lot as well? And I know Nathan, Nathan, you had kind of replied even before I replied, you know, with your own thoughts too.

So yeah. I get triggered and I don't have as much of a filter as you, you do. You're really good at that. I'm not as good, but the, what strikes me is that it's not that efficiency isn't valuable or it's not an aim. Every time I ask people, why do you do design systems? They're immediately, people say consistency. We want consistency. But then right after that, a close second is we want efficiency. And then there's the long tail of all the other reasons. It's not that it's not valuable. It's that we can't— that we stink at measuring it. And I'm a statistician by education. And I look at all the ways that people quote numbers to state things like efficiency. I'm like, there's no validity to those numbers whatsoever. Give me a break. Like, I want to start talking Bayesian analysis and a priori assumptions. I'm like, you don't know what you're talking about. But I don't play that way. Instead, what I was most triggered, Dan, by your response was, qualitative, like measuring efficiency qualitative, which made me go, what the hell? How do you—

what are you talking about?

How do you measure efficiency qualitatively? Are we efficient? Even having people answer yes/no, that's quantitative. So what are you talking about with qualitative efficiency measurements?

I am not a statistician, so, um, so I just say words that sound right and then think that it's like, yeah, that's, that's official, right? So I don't even know if I'm using the word qualitative properly or quantitatively. I'll share a story, you know, because I don't even have a good answer to your problems, Nathan. I was working with a team last week and the team shipped their first design system component and, and was doing a usability test with their first team to implement that component into a feature codebase. So the way that they had set it up was like, we have some documentation, we have a component, you know, a React component for this team to use. And they prep the team with saying like, we just want to send you these links. And if you wouldn't mind just kind of like thinking out loud with us, we just want to see like how good the thing that we made is. So the team was like, yeah, no problem, we'll do that. So the two engineers from that team were just kind of walking through like, all right, I think I would do this. Let me put this in here. And they were sharing their screen the whole time and they plugged in the component. Within a couple of minutes they were like, oh, this is great. This does what we need. I wonder what happens here. Let me try that.

Oh cool.

That worked the way that I hoped it would. And, um, and it like a couple minutes later, they're like, this is great. This is, this is exactly what we need. Thanks. And we had set 30 minutes for that meeting. It ended in like 10 minutes. And so we were like, so what should we do now? So we, we were just asking them questions and one of, uh, the design system team asked them like, like, do you think that saved you any time? And they were like, oh yeah. And they're like, well, how much time? And the engineer's like, ah, I don't know. And like, well, like. And, and so I'm, I'm backchanneling on Slack. I wasn't in the usability yet, but I'm backchanneling on, on Slack, just kind of listening in. And I was like, prompt them, like, you think it saved like a day, a week, a couple of hours, a couple of centuries? Like, what does it take? And, and so they prompted for that and the engineers were like, I don't know, like probably like a week. And I'm like, there you go. That's like, that to me is the promise of design systems, even though it wasn't quantitative. Like the answer is not, it saved us a week, it saved us 7 days, it saved us 6 days. The answer is, I don't know, like a week. And I'm like, that is the answer that in full, I don't know, like a week is the answer that I wanted because I want the team to feel like they are having efficiency. Even if that efficiency is false, I still think that pursuit is worthy. I'll mention one more thing here and then I'll shut up, which is to me, it's kind of like saying, like, I'm gonna try and quantitatively measure my health, like how healthy I am being by like going to the gym every day and walking and eating well. I don't have a good quantitative measurement, but sometimes I feel better and I'm like, I think that's good. You know, it's not as maybe as good as knowing that like, yes, I have added 1.78 days to my lifespan or whatever. Like, maybe that would be useful, but just feeling better I think is still valuable. So, you know, that's, that's definitely my hand-wavy way of going like, I think this is a worthy pursuit.

Can I cut in with a couple of things here? So I, I definitely agree that kind of subjective sense of speed is a useful thing if teams are finding that saved time valuable, which I guess is something I can say more about why I don't think that's universal in a minute. But I think something really important from my point of view to sort of add here is that we're, we're kind of using speed and efficiency interchangeably here, and I do distinguish between those two things. Like, I think efficiency I would sort of define as better use of time, you know, using your time more effectively. And the time span might be the same, actually, but like you're using your time much more effectively. You're using your time on your context-specific challenges, not the kind of common solved over and over and over again problems that we kind of specialize in solving. And so that's one thing. And I guess speed as a kind of— so speed in isolation, and this is something I felt we were on the same page about, we all seem to be on the same page about, which is that speed as a metric kind of devoid of any other context is not helpful. And I guess that's the bit I take kind of issue with because, and you know, I'm interested to hear what your kind of experience of this has been, but certainly something that I've seen a fair bit has been this kind of emphasis on speed as a useful metric in and of itself. And often that seems to correlate with organizations who from my point of view, are already going too fast. Like, their leadership is pushing teams to go faster than they are comfortable with and pushing teams to kind of get things to market as a means of, you know, experimenting in the open. And, you know, somebody's read The Lean Startup and now they just want to live in, in that world without really understanding, you know, the, the kind of, I suppose, the value of like professional judgment and experience and the ability to say, actually, like, what we should be doing is shipping our best guess, and this does represent our best guess. And so I think that's where I feel speed is dangerous, because I think it can just become sort of more ammunition for those kinds of leaders to beat teams over the head with and say, well, come on now, you have a design system, you know, what are you— why are you dragging your feet on this? You know, that, that's been one of the kind of specific areas that I've seen speed be harmful in. And I'm interested to know if that's if either of you have kind of seen similar things.

I want to jump in there, even though it's not my turn. Sorry, Nathan, I know it's your turn. I want to jump in though. I'm a little bit bullish about that, you know, for better or worse, and probably for worse, I'll admit. But like, I mean, I think that the distinction that you're making is an important one, which is like, that is a weapon for people to wield, you know, and in a lot of ways at unhealthy organizations or with leaders who take advantage of that, that's a problem. My take on that is rather than removing speed as the metric, remove that leader. Like, that's the problem. The problem is that leader, right? That person needs to go. The person who's like, we can use this speed to abuse our people even more. I'm like, get them out of here. Like, they're— and like, you know, I think we all have the liberty to do that as consultants and say that. Who knows if they would listen to us, you know, about those. But I think that's the— to me, the speed part isn't the problem. The person who abuses that fact is the problem. And that's why I'm like, I think speed is an innocent metric. I think it's a neutral metric. I think what people do with it is the important part. And if somebody is going to use that for bad— and of course, we have to agree what bad means in a given context, so that's part of the challenge. But if that person is going to use that to abuse the team, I'm like, that person is the thing that should be out of the equation as opposed to the metric itself.

Yeah, I— sorry, I just very, very quickly say, and then I promise I'll be quiet because I know Nathan's got lots to say. But like, I— I do, like, I agree with what you're saying. And I think in certain situations, it's very challenging. Like sometimes, you know, in fact, rarely I would say we're talking about one person. We're normally talking about like a group of leaders or the leadership of the organization. I think I just saw a thing from Gina popping up saying like, obviously a lot more challenging if you're working in-house, which yeah, like that's a thing too, to say, you know, get the leadership out. And I guess just the final sort of comment on that is, That I think with design systems, we're always treading this line between what's the ideal that we're trying to help people get to and what's the reality. And, you know, we want to kind of, I think, sit somewhere between those things. That's my view. And I think that when it comes to sort of the messaging that we use to sell design systems to organizations, to leadership, and then, you know, making good on those promises that we made, I think that we do play a role when it, you know, when we emphasize speed as a metric to an organization who already has perhaps a slightly unhealthy relationship to speed as a metric. I feel that, yeah, you're right. This is perhaps not a design system problem, but I certainly don't want to make that problem worse for the people that I'm, that I'm kind of working with. And I'll be quiet.

No, there's so much, so many different dimensions of this conversation we could go down with. A couple of things that occurred to me are, Dan, when you were talking about the story of usability testing, the getting started, uh, onboarding-ish process of a design system to use a feature for the first time, you know, it's almost as if the efficiency becomes invisible once they've done it and assume it. But if they— if you're dealing with assumptions in an experiment like that, What if they don't even know how to use npm? What if they don't even really know how to use React? What if they don't even know how to use a code editor? You know, all those kinds of assumptions that you layer on that are all efficiencies that really get to the root of can you take this package, import it into your project, and have a button appear on a page? And what we've— what I've seen at one team is that they started to create these experiments and they realized the people that they were going to didn't have any of those other assumptions. And so you get this conflation of inefficiencies of which the system is only a very small part. And like we're talking about, you get to the value system or expectations that you have of an org, that the speed builds in all of these different factors and really how much can the design system chip away at making the speed faster. And so I, Amy, I really appreciate you separating speed from efficiency because they really are different concepts and systems are meant to, I, or I think design systems like we work on make you more efficient? And can you tease away all of the other assumptions to understand how much more efficient are we gaining? Dan, you said a week, maybe that's give or take 4 and 3/4 days as a confidence interval around that estimate that you're— they're giving with a week. But if you say, okay, well, then people say, let's extrapolate that. Let's multiply that by 200 developers. Let's multiply that by 12 months. Oh, wow, that's $3 billion. Okay, we're not creating $3 billion, but you are creating potentially millions of dollars of efficiencies. And so how can you take those things, make sure you understand them with a strong enough sense of confidence, and then create a projection that's meaningful for some leader to make a judgment, or collection of leaders to make a judgment on its value?

I think with speed in particular, I have— and this could be anecdotal— I haven't met one leader that would say, Well, listen, the extrapolation creates an inflection point for me or creates a decision point for me where if we're going to save billions of dollars, I'm down to do this, but if we're only going to save hundreds of millions, I'm not down to do this. I haven't run into that scenario before where like, agreed, the margin of error between 4¾ days and 5 days at that scale is minimal. Extrapolated is not minimal.. But I haven't run into one leader that makes a distinction between the two anyway, even at the extrapolation point. So for me, I'm like, why are we splitting hairs between those two kinds of metrics to go like, fine, let's save hundreds of millions of dollars then instead of billions, or let's save millions of dollars instead of hundreds of millions. Like at that scale, it's still affecting change that is positive. So it feels like the wrong place to spend time on. And, you know, part of this might be the different kinds of roles that each of us have in the work that we do, you know, which I think could lead to a lot of this. It's like, Nathan, you brought up a good point about assumptions that I don't know how the two of you work. We all haven't worked together, so we don't know how each other works. I think Amy, you make a good point about I don't want to be perpetuating that. If that is a metric that can be used for bad, I don't want to be perpetuating that. I think my assumption is that we would all agree on that. How we go about not perpetuating that might be different though among the three of us where you know, you might say like, let's not focus on that metric, because focusing on that metric perpetuates it. My strategy is different, where I'm going, if I spot that metric being used for bad, part of my role as a consultant, and part of what I take on as a consultant, is to confront the leader that is using it for bad and saying, this is not the way to do this to support your team, because I see that as part of my purview and responsibility that I've taken on, which isn't to say it's the responsibility of everyone that does the same thing that I do, but it's the one that I choose to take on. So I I think how we use that is an important factor in deciding whether or not to use it or to throw it out. Yeah.

Amy?

Sorry, I'm just having a moment to think about what you said. Yeah, I think you're right. You're absolutely right. I think how you kind of deal with that is important. I think there are two sort of persistent things for me here, which are, first of all, like, I keep thinking about it now since Gina said it, but I do think the in-house experience of this is different. Like, I have found, I have experienced the in-house version of this to be very different from the version that I've experienced as a consultant. And as a consultant, I feel much more comfortable saying, you know, this leadership is kind of toxic, you need to sort this out, because I know it's at the end of the day, you know, I can, the exit door is right there, I can get out. It's not too much of a big problem. But my experience of having those kind of conversations in-house was, you know, you're a lot more tired— like, was my experience— but you're a lot more tired of having these conversations and having to kind of constantly defend your right to exist as a team and having to sort of pull together what often feel like quite arbitrary figures to kind of support work that you know is valuable and you can see the value in, but you're having to kind of support that with like these stats, which yeah, we all know there's like a quite strong element of extrapolation there. And I guess the other aspect of it is just that it is the difficulty of measuring it. Like I think that that's the other thing, like I'm— and perhaps this is where we're closer together than we kind of think we are. Like I think that What I'm not really in— what I'm really not interested in is trying to get any kind of accuracy when it comes to measuring speed. Like, it's just, I've done it, I've done it multiple times now where we've tried to sort of measure that in quite a scientific way to get this quantitative data. And it's just, it's painful and it's, and you know, it doesn't mean anything, you know, it's not reliable and stuff. So I think you're—

I wonder, can I challenge you there and say, as a like trained statistician, I'm all about like hard numbers and p-values and strong like sample sizes that are sufficient. And the reality is it doesn't matter. It doesn't— people, leaders don't care about that kind of precision. They're not caring about scientific experiments that are proving like something around a vaccine or something like that. They're just making business decisions. And I was almost appalled and simultaneously relieved when somebody in the financial office of a very large company, that one of the companies that I consult with, was like, so you're telling me the cost of the system is 7 figures-ish and the savings are, I don't know, roughly 8 figures-ish? That's good enough for us. I was like, you're a, like, a CFO-trained financier, like, get giving me approval to spend millions of dollars on people to make this system. And they're like, ah, 7 figures, 8 figures. That's all the, that's all the confidence we need. We're good. Yeah. And that's, that's the end of the conversation. So in terms of precision or statistical validity with efficiency, that doesn't matter nearly as much as maybe general extrapolations, like we're talking about, and Dan stories, like you're telling, uh, to where you can say, imagine that experience that Dan just had. With 15 components instead of 1 for 45 teams instead of 1. That's what the design system is. That's the story. Okay, good. They do the numbers and they're like, 15 times 45 times— okay, got it. That they're— why are we even having this conversation? Go, just do it.

Almost every buy-in conversation that I've had with a leadership team has been like that, has been like, they don't need to know the precise numbers to the dollar amount or to the minutes saved or anything like that. They just need to know the ballpark. And so whenever I'm prepping with teams about like, what's the executive leadership case that we need to make? It's, all right, we tried it for the last 8 weeks. This is what we found. If you extrapolate that, that gets us in this ballpark. Is that ballpark even worth pursuing at this point? You know, and, and sometimes they'll go, no, it's not. And sometimes they go, yeah, yeah, we'll do another 8 weeks of those, or that, or all right, let's do, uh, let's do 16 weeks this time. Let's double it and see how, see how it goes. And I think, you know, somebody posted in the chat, like, enough low-fidelity information to make, to make a high-fidelity decision. Uh, one of the, the the designers on my team, um, that I'm working with now, um, has a great phrase that they kind of use. They say the MVP of alignment. Like, oh, what a great— what a great way to think about it. Like, we don't need to be fully aligned on this, we just need the MVP of a lot. We just need to be aligned enough to make a decision to take it to the next step. And I think a lot of design systems work is, is that, um, one thing that Amy, you said that I 100% agree with is I am leaning into my bias as a consultant. So I'm not at all recommending that, oh, the way that I do it is how any design system person should do. Absolutely not, because the context is so different. If I was an in-house person, you know, working in-house, I would have a completely different strategy than I do as somebody who's working as a vendor or a third party or consult. Like, they're just different roles and they have different sensitivities and different respect levels, you know. And I think— I do think that it is worthwhile for consultants to act a little bit more like in-house folks when they can, and for and ask folks to act a little bit more like consultants when they can. There is value in all of those things, I think, if only for empathy, but also because there's value in each of those positions being there. So certainly, I'm sensitive to the fact that if you work in-house, there's stuff that you can't say that a consultant can say, and vice versa.

Yeah, yeah. I think that's right. I think that sort of— Something that I wrestle with here is that— and I think this is maybe a really important thing to, to say about the context that I'm thinking of when I say speed is not a good metric. So a lot of the organizations that I talk to, one of the things that they're struggling with and that they want support with, and they're hoping a design system will help with, is that the quality of their outputs is a concern, like their maturity is— like design sort of experience and maturity is not great, or there are sort of very heavy-handed product owners that are just kind of putting a lot of pressure on their teams to sort of emphasize delivery above any other measure of success. And that's often the context in which I'm working or thinking about or supporting people in. And I think that for me is where speed becomes this really kind of toxic thing, because actually, like, making them go faster, making them ship bad stuff faster at scale, it's not like— it's not something I really want to do, honestly. And I think, like, I think that's maybe a really important piece of context to introduce. Like a lot of the time, I'm not thinking about or working with organizations who are kind of particularly happy with the quality of the user experiences that they're providing. And I think that if a design system is, is to play a role in improving that quality, and I know I often fall on a different side of the coin to people about this, but, you know, I think that's something a design system should be doing.

Then like Amy, with quality in particular and Dan, the two charts we did in Twitter, I can show them side by side.

I was just thinking that. Yeah.

Yeah.

But like, there are all sorts of different things you can measure and they express what you value out of your design system. And people say consistency and then they say efficiency and then there's this long tail of other things. But I always find that the word quality doesn't start the conversation, but it ends the conversation. It ends the conversation of what they're actually trying to get out of the system and ends conversations when the system's is not giving them enough quality out of what the system provides. And they're like, not good enough. We're going to do it ourselves. And the— Dan, I was struck by the fact that you had quality low in terms of— very low in terms of importance to design system success, whereas I had it mid-range high and it really is trailing consistency and efficiency. So first of all, we have to talk about what do you mean by quality? Like, is it performant? Oh my gosh. That's a whole different challenge. Is it accessible? Yeah, you need to think about that. Is it usable? Is it all those things that end in -ity? Usability, accessibility, etc. All those different things are all munged up, and even consistency is kind of munged up into that. And so really, Dan, I'm curious, why did you rank quality so low? And what does that say about the outputs of the systems you work on?

Yeah, yeah, fair, fair point. This is, this is maybe my hot take for design systems is like, I'm a designer who went to design school. I also started before I was a designer as an engineer, right? So that's my background, that's my wounding, you know, like for better or worse. Um, I have seen quality not matter like too many times that I'm like, why are we chasing this? And consistency, same thing too. I'm like, I've, I've seen teams chase consistency so much. And again, this is anecdotal. This is not a recommendation or a suggestion for anybody. This is my experience. I've seen teams and been on teams and myself chase consistency so much and in that and never achieve it. And even in points where I've seen teams get close to achieving it, it didn't matter. Like, didn't matter at all. So who cares about consistency? Like, and again, that's, that's my hot take. I'm being a little bit flippant about it. And same, same thing with quality. And part of it is what you just mentioned, Nathan, which is like, how do we even define that? You know, it's hard to find even an MVP of alignment on what quality means. Usually it means something that I personally don't relate to quality. And also, if you look at, if you look at the state of products in the world that are out there, most of them are low quality depending on what your standards are for quality, but they're out there and people can use them. And I think that that is the important part of product design. Like, my part of my belief about this, and, you know, some teams that I work with are self-selecting in because the teams that I work with believe the same things that I believe, otherwise we wouldn't be working together. The teams that I work with, I try hard to convince them that like quality is a thing that we can grow into. Some other things are not things that we can grow into as easily. So if we're going to defer some things, let's defer quality because we could always iterate to that. That's like, that's what I believe. I believe strongly in Agile. I believe strongly in product design being an iterative, iterative model. We can always iterate our way into better and better quality. We'll never achieve quality but the pursuit is worthy, we could always do that over time.

If—

but what is harder is adoption. In my opinion, adoption is the hardest thing. So let's focus on adoption first. Let's focus— and with a lot of teams that I work with, and the deprogramming part is the hardest part, because they believe if we just make the better thing, everyone will use it. I'm like, well, what— how do you define better? And does that align with how other teams define better? For a lot of other teams, it's like, can I use it is better. Right? And that's why a lot of teams start with Material Design or Bootstrap or, you know, something else that's not their own better thing. And that's why design systems that are objectively better fail a lot of times because, because they're not as easy or they're not as understandable or like, and quality is usually not one of those things. So that's why I'm like, if we could just get a blue box integrated into, into a feature team's codebase, we can iterate that blue box into the best component ever. Over 4 versions. Like, that's what people are already good at— engineers and designers, they're already good at making things better over time. What they're not good at is adoption. That's the thing that needs to be trained and practiced and modeled and all that. So we start there, and something's got to give. And in my opinion, quality is the thing that gives, because you always get— you can always get that back. Again, hot take perhaps, but that's, that's generally my stance on it.

I like how you characterized the gradual emergence of quality. And when quality costs money, and when we think about test— you assure quality through testing, and you think about, okay, we're going to check it on browsers and devices, we're going to do code linting early on, we're not doing VRT with the first release, we're not doing our visual regression testing, or other kinds of more sophisticated testing capabilities. And that's an expression, I think, to me, Dan, of what you're talking about. But quality is always a balance against speed. The more you test, the more hoops you have to jump through, the more things you have to check, the slower you're going to be able to produce something that other people can adopt. I also totally agree with consistency. It sometimes feels like a total fool's errand, even usability, from the perspective of that all comes through composition. And we don't control the composition of individual product experiences, nor the pressure applied to those product experiences to all have common navigational schemes, to all have IA and crossover journeys that work well together, that have good handoffs, that comes through those teams connecting. That's not something, frankly, I have time to, to moderate or to force as a discussion. And so to me, I like how you characterize, and I too say let's gradually build into higher and higher quality of things, and they're never going to be perfect. You're always going to have things you need to fix and grow and improve and polish.

Amy, what's your take on quality?

Oh my God. I think it's different from both of you. And I think it's— we've had 30 minutes. So I think we need to do a series about design system metrics and the next one will be focused on quality. Maybe. That's really good. Yeah, no, I definitely have more to say than I can say in minus 1 minute. But this has been very insightful and I don't really know how to wrap this up, given the informality. I'll wrap it up.

I'll wrap it up. I claimed being the organizer, so I'll wrap it up.

There you go.

A couple— 3 things to point out. One is somebody asked in the chat about, has anybody turned these into KPIs for a design system? Nathan has a great article on Medium. It's a couple years old, Nathan. I don't know if you have an update to that or think it's irrelevant, but great article about starting points for that. So that's a good starting point. Gina also mentioned in the chat that her talk at an event today is about quality. So if anybody can and wants to join that, that's probably a good one to do. And then last thing is that we had talked about doing a part 2 if needed. It seems like that is needed and interesting to folks. So we will organize a part 2 probably a week or 2 from now. We already have a tentative date, so we'll confirm that. And then I don't know, try to do one, try to do another one of these and maybe involve some, a few other people too.

Nice.

This was fun.

Very well done.

Thanks everyone for doing this.

Yeah.

Good to see you both.

Yeah.

Hello. We'll see you soon.

See you soon.

Bye.