The Role of a Product Manager: Everything and Nothing - Josh Elman at Mozilla Speaker Series

Thank you.

Better. The most important thing that I learned was to think about our products through the eyes of the people we're building them for. And so when Diane offered me the opportunity to introduce Josh Elman today, I jumped at it. Because I think that not only am I excited to hear what he has to say, but I think that the things he has to say are incredibly important to all of us in the room and all of us watching online. Building products for the people or thinking about products for, through the eyes of the people we're building them for, is an essential discipline. And it's one that matters to everybody, whether or not you have the title product manager. So, it's with great pleasure that I welcome Josh Elman to speak to us today.

Thanks. Thanks, Justin. Thanks for having me over here at Mozilla. It's great. I'm looking around this great room, and I know many of you guys are watching on the camera. So I've never spoken to this big of a virtual audience before. So I'm excited, too. And it's great to see friends like Chris Beard, who spent a month, you know, got to spend a bunch of time when he was hanging out with us over at Greylock. And then I work with John Lilly very closely, who I'm sure many of you know from his Mozilla days. So it's always fun to be here and, and just talk about product management. And really, in talking about product management, Product management is a role, and that's kind of what I'll talk about a little bit, is sort of the role of product management. But really, product management, the whole role of product management, is to be part of a team that builds great products. And so most of what I'll really talk about today is maybe what a product manager does, but really what a product manager does in the context of a team that's thinking about how to, how to build great products and work on it. So, you know, I really appreciate Justin's introduction. But you're probably wondering, you know, who is this guy and, you know, what has he done? And I started as an engineer. A Windows engineer back in 1997 at a company called RealNetworks, if you guys remember RealPlayer. And I got to build a lot of that. That was sort of, at one point, the future of audio and video on the internet. And just coding away features. And, you know, within a couple years, I was so excited about what we were building and how we were building it that rather than just being kind of working on my own features, I got excited to go sort of help lead a team and sort of ended up running a team of, you know, 15 or so engineers, program managers, you know, on the core client, and really just loved this idea of, like, how do we ship this thing that goes into hundreds of millions of people's homes? I once went to one of my 60-year-old neighbors, and they said, oh, what do you— what'd you do? And I'd just moved down to the Valley, and I was like, oh, I used to work at RealNetworks. They're like, RealNetworks, we love it! And they opened up their living room cabinet, and inside of— they had a computer that had their entire music catalog all organized in RealJukebox, which was a product that I had worked on, and it was RealPlayer. And Rojukebox. And they had organized all their music, and they were using it to, like, play through their sound system back in 2003. And, like, that's the moment we all live for when we're making stuff, is, like, normal people, not just tech people, using our products to, like, live their lives even better. And that just sort of, you know, helped catalyze kind of all the work I'd done for 6 years at RealNetworks and just kind of kept me going in my career. Since then, I've gotten to work on a bunch of products that hopefully you still use. Maybe if you don't use RealPlayer, like LinkedIn. I was, um, joined LinkedIn. It was about 15 people, and we were figuring out how to build the future of a professional network. And we did a ton of work on growth. I got to launch LinkedIn Jobs. I worked at a company called Zazzle, which does on-demand custom products, and this idea that anybody who's creative can upload a design and have a storefront available. And we called it Bits to Atoms because we actually had a factory that would take your design and actually print it on a physical product and ship it out to you in 24 hours. So it was kind of this really neat digital stuff that we were doing all the time. Then you go to the factory. And I once went to the factory and realized there was like 100 shirts all printed with like, with this bizarre default font. And everybody was just using the default font and it made the shirts look terrible. And it was like this like totally different realization that these like dumb little software decisions you make like affect these physical things that then get shipped out. And you have people walking around with like words just written in this like horrible, weird font as opposed to even just something like simple at the— So we spent a lot of time on, like, good fonts that printed out well to find our better default and a better set of suggestions. I went over to Facebook in kind of early 2008 as the— I just saw the Facebook platform blowing up and thought that'd be really fun to work on. And if you guys remember Facebook Connect or sort of the heyday of sort of all the apps that were built on Facebook, I was kind of— got to be right in the middle of all of that. I helped launch Facebook Connect and brought that out to all the different early partners that worked on putting Facebook and making their own site social. And as much as I liked working on the platform side at Facebook, I got really excited about working on a product again. And so I went and joined Twitter in late 2009. It was about 80 people in the company. And the challenge I worked on is probably similar to what I think Justin said he's working on now, which is we had the problem where everybody signed up, but nobody stuck around on Twitter. And they said, help us fix that. And, you know, we got it from about 10 to 100 million active users in my time there. You know, they've, they, people would still say that's a problem with Twitter, that a lot of people sign up and don't stick around, but I'm still proud of how much better we got to do. But, but in all these roles, you know, the one thing that, that really, you know, you know, that, that I really experienced consistently was, how do we as a team build and ship great products together? And as I kind of took the role from engineer to product manager, and then was really a product manager most of the rest of my career, until I'm doing this VC thing. I really thought, like, what actually is that job of a product manager? And so, I put together this little discussion. I wrote a blog post and then put together this little, little talk to kind of help try to explain the role and think about how do we define it in a way that really enables the team to be so successful. So, you know, I think, you know, I've always had this question. I think a lot of people do. Like, what exactly does a product manager do? And, the worst thing is, can you actually explain it to your mom? 'Cause my mom was like, what do you actually do as a product manager? Like, I understand your brother. Because, like, he's an engineer and he writes code. And then he became a designer and, like, you know, he makes designs. Like, I understand what he does, but, like, what do you actually do? And so, I first thought that we define product manager by what it doesn't do. 'Cause I think it's really important. Like, a product manager doesn't write code. The amazing engineers on the team are the ones actually writing and responsible for the shippable code that goes into the product that actually gets delivered to the world. Product manager doesn't create mockups. Now, you might create wireframes. Sketch things out. But those final designs, those final renderings, the visualization of the product, really, you want to have great designers on your team that own that, that think about how to make things pixel perfect and have the interaction flows be exactly right and really experience what the user is doing. And a product manager should understand that, but doesn't need to actually be able to do it. I personally got Cs in art in school and could never draw anything. So my whiteboarding skills are almost square boxes that people, that great designers have always done a much better job of. Product management also doesn't sign deals. It's great to have a business development team that's out there negotiating real partnerships with companies when that matters. Product management is often involved because there's a lot of implications of the deal. But, you know, the business development team actually owns and signs that final deal. Product management doesn't do PR. They're often, you know, involved in the messaging, involved in the voice, but, you know, hopefully have a communications or a marketing team that really owns That press release and other things. So, you don't really own any of these artifacts. When Reid Hoffman interviewed me for LinkedIn, his, the question is, one of the best interview questions I've ever had, was, you know, each of these other functions has very specific artifacts that they own. He's like, what is the artifact of a product manager? And I struggled a little while to answer the question, cause I went through this whole thing. Well, it doesn't own any of these. And realized the artifact really is the vision document and the spec.

Right.

That sort of everything builds off of, and capturing that, whether it's a spec or notes or ongoing iterations, is really the one artifact a product manager creates that hopefully the rest of the team is able to do their best work on. So, you know, one way to think about a product manager, as I said, is if it doesn't do all these things, it sits in the middle of all of them. It sits in the middle of all the communications. It sits in the middle of defining the requirements and roadmap. Figuring out who the competitors are and understanding their capabilities. So that we understand what we're building and what we're not building, you know, inside of all that. Understands defining the problem and the value proposition of the product. Figuring out the launch timing and any sales or marketing collateral you need to do with partners. Figuring out how to define the market and customer. Being the evangelist. So it kind of sits in the middle of all these things, but again, they're all pretty nebulous. And it's hard to use any of these as, like, the definition of a product manager. If you think about it simply, you can think about it as just this intersection of of user experience, technology, and business. You know, right in the middle, the product manager kind of sits in there. But again, none of these things really define what it can do. So I tried to figure out if I could catalyze this into at least a one-sentence description. And what I came up with was, a product manager helps their team and company ship the right product to their users. And I'm gonna use the rest of the talk to kind of define what each of these pieces mean. But, but if you really think about it, like, If you didn't have a product manager, the whole team and company might actually ship the right product anyway. Designers, engineers, product, you know, business development people. They can all work together really well and ship a great product. You don't necessarily need a product manager. And what I've seen is the cases where you have that product manager who does the right things to help and makes the team better, you can have 10x, 1,000x better efficiency, effectiveness, delivery of the product. But, I've often seen cases where the wrong product manager makes the whole thing a lot worse. And, like, instead of helping, they're doing the opposite. So, so it's still a pretty nebulous job and it's hard to find ones that are great. And when you do, hold onto them very tightly in, in your company and everything else. Because, if you have the right one helping the team, it makes everybody so much better. And it actually is a full-time job, or more than a full-time job, when you have multiple product managers, to make sure that the team is most effective. So. To kind of start with, with this, I always like to think about, like, what does it mean to actually help your team? Who is your team? If you're the product manager, who are you ultimately responsible for catalyzing and helping to ship the right thing? And so, you know, when I think about a project team that's delivering something new, it is the group that is working on a product or a specific feature, you know, in a concise area of the product, where their goal is to deliver this out to consumers. And so the product manager sits there with the designers, engineers, QA, often documentation and marketing, on one functional group that says, together, we're gonna ship this product. Now, you might actually, I think functional, I didn't mean functional group. I meant sort of one project group. Because you might have different functions and they might report up in different structures. But when you're working on something, that's your team. Because it's, each one of you together is collectively responsible to deliver this thing to the market. And if you guys don't do it, it's not like somebody else in the company actually will. Like. It's really on you. And, and, you know, what I like about teams is, we always had some other groups that were sort of associated with the team, even if they weren't full-time. Whether they were from business development or support or legal, where they were really responsibly part of our team actually delivering this project. So what is this product manager's role on the team? A lot of people have said that product manager is the CEO of the product. I don't know how many times you've heard that. And whenever I meet MBA students, if any of you have MBAs, sorry if I'm offended. But whenever I meet an MBA student, they're like, what do you want to do? I want to be a product manager. I was like, why? They're like, because I want to be the CEO of the product area. And I look at them and I'm like, OK, but that's actually the worst possible thing you could do is, like, that's the backwards way of thinking about it. You're not the CEO at all. You don't own the resourcing. You don't own the, the P&L for the product when you're a product manager, especially at a technology company. But you are the team leader. Your, your job is to kind of be the one who helps set the cadence. Brainstorm everything effectively, manage all the operations. Like, you have to make sure the team runs. But I like to think about the product manager more as, instead of the person in the front of the raft, like, paddling away and, and saying, we're going here, it's kind of the guy in the back who just sort of helps course correct and adjust and steers it just right so that you don't hit things in the river. And most of the rest of the team is really, you know, you know, doing the work. But yet, you know, he's wearing, he's got all that stress of, like, we better not hit that rock. I see 4 rocks coming down the way. Like, oh, we have to take this fork. Like, you're always planning all of that stuff, and you're helping the team sort of, you know, keep the cadence from that position. So, great product managers, you know, and this is the stuff that I really liked on our team, you know, really figure out how to set the cadence. And I think this is the most important thing for any team, is how do you just collectively continue working and being more productive as a group. The way I used to put it was, you know, we used to build the roadmap sort of quarterly. I called it sort of brainstorm meetings. But every quarter, we'd come in with a set of problem statements. And I'll come into this in a sec. But I'll talk about our, like, we'd come up with these sort of problem statements. We'd figure out how to build the roadmap together, rather than ever feeling like the roadmap to the team was, was handed down. Because that way everybody really owned it. But the product manager's job is to take from that and then articulate it really, really clearly and consistently. Like, it shocks me how— How many people sort of, like, throw a little roadmap up as a list of bullet items, or throw it up as sort of, oh, we'll have a roadmap somewhere, and it's on a PowerPoint slide if you saw it in a deck at one point, versus really figuring out how to make the roadmap a living, consistent document that other people outside your team can really understand? I found that holding regular, like, weekly meetings, one week, one meeting a week for the team, that was sort of a cadence meeting. We called it product operations. help the team be much more effective without all the meeting chaos that can then ensue. If you have a planning meeting for everything, or a strategic meeting for everything, or a check-in meeting for every little thing, you can get so overwhelmed with meetings that you don't have time to do actual work. So we tried to consolidate it all into sort of one. And often this was biweekly, not even every week, in order to kind of just keep the, the cadence of our meeting. And I'll walk through what we did there. We had this thing that we called ACT Solid, which was, we felt, was the product manager's job to work with all the other teams outside the company. And ACT Solid was an acronym that represented a bunch of the teams that you would have to work with that were outside your design, engineering, and product resources. And then, lastly, and probably most importantly, is take and share clear meeting notes. I'll come back to this, but I found that a product manager's job is writing the notes even more than writing the document up front. Because the conversations and collecting that and consolidating that into something really that became the living document and capture of what was done is so important. And, again, something I think a lot of people sort of don't spend as much time on. It's like, oh, well, we were all in the meeting. Of course we came out with total consensus. Which, yeah, never happens. So in terms of our quarterly planning meeting, this is something that I thought was really fun and useful for us at a couple of different companies that I worked at. You know, we would come in with, with, I'd come in as a product manager and we'd been talking about and brainstorming different ideas. And everybody always has a feature wish list. But every quarter we'd be like, what are we gonna work on this quarter? How are we gonna drive impact? And we would understand, hey, we, you know, if I was on the growth team, we need to figure out how to make this grow. And the couple areas that we identified for the biggest holes are this, this area where we're not converting users who are signing up to stay active enough. Or we're not getting enough users to find their friends. Or, We think, you know, a couple of the problems are people aren't even aware of it. We need to figure out some new viral campaigns. So we'd kind of come in with a bunch of questions and ideas like that, and then we'd let everybody pitch. Everybody on the team got to pitch whatever features or ideas they thought could be most important in order to achieve that goal. And I tried to make it really open. So if you were an engineer, if you were a QA person, if you were a designer, if you were, you know, from legal who was helping us out, Everybody had ideas of what we should do to achieve the goal. And then we made everybody vote. And, and you just had to pick your top 3. You were like, I think if we do these things, we'll have the highest impact on the company this quarter. We should invest in these. And whether it was quarterly or, sometimes they were built up for very long, you know, even longer projects. But, we'd have everybody vote on what they thought was gonna have the biggest impact for the company. And I like to think that everybody on the team is pretty invested in the success of the company. And just being told what to do isn't as fun as as being part of the discussion. After everybody voted, it was non-anonymous voting, so you had to put your initials up there. We would go through and, and have a discussion of why. So if one person voted for something but nobody else did, they could make an impassioned case for why they thought that was one of the most important products to do that quarter. And if a bunch of people voted for something, they could make a case for, like, everybody should have voted for this. Then I let everybody re-vote so that you really kind of felt like, OK, well, the trend is moving towards this thing. I want to make sure this one actually wins. Or people might shift their votes around to kind of, you know, really commit to their stuff. And then we really would take the top 3 out of that whole, that whole area, and that became sort of our roadmap. And it sounds like, oh my God, you can't have consensus voting in order to deliver a roadmap. And I'm, you know, I can imagine in an organization like Mozilla that, that might be crazy. But for me, I always found that it got everybody really invested in what we were doing. And even if your project didn't end up winning at the top, sort of the disagree but commit philosophy was, was more important than, than just kind of top-down by edict. We must work on this right now. There was always some adjusting, and, you know, my team leader on engineering and I would always kind of be able to shove one of our projects in if we needed to. But, but, but we never put it in the, the first position, cause we knew that that was really what the team had done. And this was, this was sort of how, as a product manager, we kind of let the whole team figure out what to do. And on growth, it ended up working really well, cause we had so many projects that we had to do to kind of drive growth. You know, the short answer is, which one should we do? It's all of them. So how do we kind of get some of them done this quarter or this next 6-month period and prioritize which ones we do sooner? You know, in terms of, of the more weekly or biweekly cadence, you know, as a product manager, we try to set up a way in which we really help the team be more effective. This was our meeting that we'd have every week or two. Was one, you know, I would share any company news that was relevant, just cause it's, you know, if I was sitting in a bunch of different organizations and understanding what everybody else was working on, we'd kind of make sure the whole team hears that. And if you, you would have this place to hear it so you wouldn't need to go to lots of different meetings to try to get status on other things. Anybody who was launching something soon, and, you know, at the companies I worked at, Twitter and Facebook and LinkedIn, we were, we were launching, you know, every couple weeks we were, the team was launching something new, whether it was a small A/B test Or something bigger, we kind of go through a final gut check, have the whole team able to see it and review it. The thing where we spent a lot more time that I thought was really important that I, I regret not seeing a lot more teams do was learnings and analysis of recent launches. We always did a look back. About 2 weeks after something launched, people who launched it had to go pull a bunch of data and say, we expected this kind of impact. Here's what we're seeing. Users who are behaving this way, you know, are doing this more. Users who aren't are— You know, we, but we aren't getting enough users to do it. We think we need a new feature that will drive more people to try it. Whatever it was, the lookback became actually one of the most important ways that we were able to manage the group, because everybody was able to, to figure out, like, did our things work? So that the next time we were able to actually say, hey, what did we learn last time? That feature didn't work. That one did. And, and I would just highly encourage you, if you don't do lookbacks, you know, you should do them. You know, and often, like, 2 weeks after a product launches is a really good time where you have a week of data and then you spend a week sort of starting to look at at least the first week's data. And then try to do it, like, 2 months later, too, because then you've had enough time for something to, to settle in and you see if it still matches your patterns or not. And hopefully a bunch of the queries and other stuff you had to build in that first week or 2 are still useful and are easier to run the second time. We'd often do a roadmap check-in, so anybody who's working on something new. And then, you know, a guest speaker, too, kind of like your monthly speaker series here, just to make sure that the team felt good. And so with this kind of one meeting a week that was about a 90-minute meeting, and as I said, often I did it every other week when we weren't in, like, super crunch mode time. Everybody felt checked in and we didn't have to have a lot of the other meetings that I'd had at other organizations. But, this is the product manager's job. is taking the notes. And, you know, setting up the meeting and really taking the notes. And, and the job isn't that glorious. It's really documenting. I used to spend longer after a meeting writing up the summary of the meeting and the analysis of the notes and gathering everybody's decks, you know, if we had, like, presented a deck on something, and writing up kind of our summary, taking it together, any action plans, than I ever spent in the meeting. If it was a 90-minute meeting, I'd probably spend 2 to 3 hours either you know, the rest of that day or that night or, you know, the next day to get the meeting so consolidated, the notes so consolidated that kind of everybody knew what it was. And if it was any brainstorm meeting or any project planning meeting, I would spend a lot more time doing this. And I found, and then we actually would send it out to most of the company. And by doing it that way, you know, everybody got involved. And I just find, as a product manager, that's sort of the crux of the job is helping the team get on these cadence, helping ask the big questions. work through these problems, and then communicating it back out afterwards. The other thing we came up with, and this was, this, a group came up with this at Twitter that I thought was just useful, so I use it all the time. It's called ACT Solid. It also, another acronym was Lost Acid, but we thought that ACT Solid was better. And these were all the different teams that anytime we shipped a feature, everybody in these various groups that were not part of the day-to-day product organization had to be aware of it. And, you know, everybody thinks, like, oh my gosh, legal. They're usually a bottleneck. And we found legal was actually an amazing enabler. When we looped legal in on something we wanted to do that was somewhat controversial, like person— like after I left Twitter, a bunch of people worked on this thing where they personalized your Twitter signup experience based on what websites you've visited before, because they knew that from your cookie data. There was a lot of work with legal. And legal was not like, oh, you shouldn't do this. It's a privacy violation. Legal worked with them to actually set up the right policies, set up the right disclosures, to really make it work, and the legal was a great partner in that. And so all of these teams, when you loop them in early, customer support, we may increase some of your customer support needs with this feature, but the reason we're doing it is for these real reasons for the company. It actually works much, much better. And so we called it Act Solid. If a product manager went around the rest of the company and everybody felt looped in, you could actually get graded as a product manager by these other teams on whether you were somebody who acted solidly or not. And so that was kind of this whole idea of, like, how do you really just help your team, define your team and help your team get successful as a product manager. And you can see how much is really dependent on this person just making the team better. And if they're not making the team better, if there's too many meetings, the notes aren't written up well, the questions aren't clear, people don't feel involved in organizing the roadmap, then you're not really helping your team. So the next thing I like to think about is the product manager's job is not just to help the team, but to help the whole company. And, and the, the product manager really is in such an important position to be able to be part of the rest of the company's, you know, overall organization, understand the product strategy, understand the business metric strategy of the company, so that they can take that whole focus and figure out how do we translate this down for our team. How do you take the company's overall goals and objectives and tie that to what our team's working on, and to make sure that what our team's working on is in service of those goals, too. The team, you know, as a, as a product manager, I never think it's my goal to say, our team's so important, cause, cause I believe we must do this. But really to tie this back to the company goals, the CEO's vision, the founder's vision for why the company exists, and are we really, you know, adhering to that, inspiring that, you know, taking it to the next level. I think it, there's a lot of times where I've met product managers who get so excited about the thing that they're doing, they get very myopic. And they forget that it's in the context of this organization. That everything I ship, you know, depends on the brand. And, like, well, I'm sick of this old feature, so I'm just gonna do it my way. You know, our team's just gonna do it our way. And, and you miss, you miss that really important connective tissue that ties everything together. When I've interviewed product managers, I, I really look at this filter as one of the most important to kind of tell the ones that I think will be great for our team or not. And it's how much, when they're talking about, I was like, well, what did you build in the past? Why'd you build it? Like, how did you like it? How many times as they're answering those questions, they're tying it back to the bigger vision of the company? They're tying it back to the con— I had this conversation with our CEO, and he inspired these couple of things. And then we went and came up with this really creative solution that was, like, really amazing and really delivered this value back to the company. That's a great answer. I've heard often, many times, where, like, well, The company was kind of struggling, and so I came up with this thing that was gonna revolutionize it, and our team went and built that, and like, screw the rest of the company. And that's not a great answer as a, in an interview. But I've seriously heard it many times. And if anybody here was from Yahoo, I apologize, 'cause a lot of them were people from Yahoo in the later ages. I just was distinctly remembering one conversation. And so, so that's a, it's just a tricky It's a really tricky balance, but the great product managers know, you know, why we're all here and why the company exists. The next thing, so, is help your team and company ship a great product to your users. So, I've, I just called out the word ship, cause you can't emphasize enough how important it is. If you can't help your team actually put something in customers' hands, you're not a good product manager. Even if the team loves you. Even if the team operates really well. Even if it's, everything's tied to the company's vision. But if you can't actually get it out the door, Then, like, you know, they're, and by the way, the pressures from a great team are often not to ship something. A great designer doesn't necessarily want to ship it until they feel like the design is whole and complete and perfect. A great engineer doesn't want to ship something necessarily until it's perfectly robust. And by the way, those are both right. Like, you can't, you know, you gotta ship it. But the product manager's job is to, like, push and pull on everybody to create the set of sequence. That means, like, how do you actually figure out the thing that is good enough to ship to our users today, that won't destabilize the company, that won't destabilize the product stack, that won't cause a ton of new customer support, a ton of new issues, but will actually put it in customers' hands? And it's never everything. I mean, we, we forget on our iPhones now, but they did not have copy-paste in v1. And you were going back between email and web browsers and, and all these things all the time. And you literally couldn't copy-paste anything, and it felt like A total pain in the ass. And you're like, how could Apple have shipped it without copy and paste? And they're like, because they had to ship a phone. They hadn't figured out what to do with it and build a clipboard into the operating system yet. And they decided that it was better off to do it. And you know what? I don't think any of us regret the fact that, you know, a couple versions later, we got copy and paste. We, the, the funny thing about when you ship without something and then you add it later, nobody remembers when you didn't have it anymore.

Right.

It's like, the moment the feature's there, It's immediately eliminated that we ever, you know, struggled with, without it. And, and new features just sort of do that. You know, on the downside, once you add a new feature, you know, it changes people's perception. If that's a bad feature, you, you know, you, your product now is worse or more bloated because of it. But I think a product manager's job is to sit and just be this, this perfect tension, grinding right in the team to, like, here's what we need to be for launch. Here's how we make these, these really tough trade-offs. I like to say that great product managers really understand this balance between getting it right and getting it out the door, because you can't always do both at the same time. But, but once you ship, it's how do you, how do you ship the next things and, and do it right? And by the way, the other thing I learned is I see a lot of teams, like, have a whole bunch of stuff they want to ship, and then they make all these trade-offs, and then they ship, and then the very next thing they do is go build all the things that they cut on the, the floor right before they shipped. And it often turns out, when you actually put it in your users' hands, they don't even care about 5 of the things you were about to ship. They actually care about these 3 new things that you hadn't quite identified as a need that are actually broken right in that initial, right in that initial moment. And so you actually learn more by shipping, learning, figuring out what to go work on now, not just, OK, we shipped. Now let's just go back to our list and keep on plowing along. So I think it's a, it's a really important thing to kind of Ship, listen, learn, and keep doing it. So, look, we can talk about shipping and a team and building the right thing, but it only matters if you build the right product for your users. And so I'll go through kind of how I've thought about the right product and figuring that out, you know, sort of as a product manager and how I've seen great product managers do it. You know, the first is, I call it believe but listen. Because, like, you have to be creative. Like, part of what we're doing in our industry is building something that's never been seen before. You know, we're building tech, like, the power we have in technology is we get to build this thing that nobody's ever done. You know, and if we're just building a clone of something else, it's really not that interesting. So, so you really get to think back and like, OK, we have this great idea. It should be amazing. And then how do you actually figure out if it's a good enough idea? And, and I think like, the first thing is, you talk to as many people as you can. And, and everybody's like, well, should I keep it stealth? And like, yes, sometimes things are are great to keep stealth within a company or in a very limited situation, because you have a magic moment when you deliver something new to your consumers that they weren't expecting that really meets their needs. But you certainly can't do it without talking to people and, and talking to people that you trust. A lot of, I, I love to get ideas from testers and active users. And the feedback that you want to get is not, I wish the product had this. I mean, everybody knows this. It's, how do you figure out what problems they're having and then how do you get really creative about how to solve them? I look a lot for criticism from non-users. You know, when we were trying to figure out growth at Twitter, we spent a lot of time trying to figure out, like, why don't people use Twitter? And we found one of the best sets of people that we could ask were people who had signed up, left, and then came back, and then finally got it the second time. Because we found that we could actually get them to really articulate what didn't happen for them the first time they signed up and what did happen for them later. So that we could figure out those gaps of where our messaging and our first experience didn't deliver it, but something else in the world kind of recatalyzed them onto Twitter. And that was super helpful for us. I'm a big believer in getting input from founders or leaders, the CEOs of the company, because they, they see so many more things at such a different level that it often helps to kind of just have their spark and their gut reaction. But if they're like, oh, move the button here and do the button red, like, that's useless feedback. They aren't in the weeds enough to really know what to do with your product. But if they're like, wait, this flow doesn't quite make sense. What if the messaging was more like this? Because we're seeing this and this happen in the world. I think that's really, really useful feedback. Because that will often change the way that you're thinking about it. And they're seeing stuff from perspective that, that you might not be. And then, you know, ideas anywhere from, you can get them. I got really good at reading Twitter to figure out ideas for what we should do for Twitter and to figure out how people talked to Talked about Twitter. It's, it's great to have this public water cooler for that stuff. You know, you guys have such a great community at Mozilla. I'm sure you get tons of ideas from them. And it's, you know, how do you filter, how do you filter it down is the hard work of product management and a creative development team. But it, but it is really exciting. And the second thing, and we talked about this a little bit before, is how do you measure results? How do you actually figure out if what you did worked? And I think a lot of people—

Yeah.

ship this stuff. And they're like, it's gonna be great. And then you ship it, and then you're like, OK, did that work or not? I don't know. Let's just go on to the next thing. And I think it's 'cause people don't think about it like scientists. And yet, as creative technologists we are, as we're building products, we're basically some form of, like, actual scientists or social scientists. I like to call this, like, the data scientific method. It's, start with a hypothesis. Come up with the thing that, we think by shipping this, we're gonna cause this behavior, and these types of users are gonna start doing this instead of what they were doing before. Figure out what those metrics are ahead of time. Then when you run, when you actually ship the product, you're basically running an experiment. You're running an experiment on people, on your users. You're saying, with this new feature, with this new button, with this new tool, with the way that we've marketed it to you, do you now operate in a new way? Do you now behave differently? And if so, you're gonna have a much better sense of, of, of what you're doing. You know, when it works and you're gonna get all that data back, and then you're gonna be able to, then you're gonna be able to generate your real hypothesis. Awesome. This worked. This feature worked. We should double down. We should open a new business line. We should build more products like it. OK. This one didn't work. Did it not work because we built the wrong product or because we had the wrong hypothesis up front that users didn't even want to do something? You can go back and figure all of that out. But if you don't start with a hypothesis and you don't then get any data, you're screwed. And then, and then the last thing is always keep an eye out for unexpected learnings. There's always that, that really cool thing you didn't expect to happen, you know, when you make a set of changes, that if you can figure those out, those might be really cool threads to, to get really creative about. The last thing, and, and by far the, you know, but probably the most important, is fundamentally your users. And as a product manager working with your team, Figuring out what to ship. How do you make sure that everybody's thinking about the users all the time? It gets so easy to say, oh, well, if we just do it this way, it'll be, like, so much easier and we'll get it done much faster. But how do you kind of remember, wait, but actually, every time you make a product, there is some user, some, you know, young 15-year-old kid somewhere in Iowa who's opening up a computer and is getting on your, or their phone, and getting on your product and experiencing it in a certain way. And it gets very easy when you look at data, and when you look at all the different big conversion rates and MAU/DAU charts, that you forget that, like, all those things are summed up from, like, one individual person in their lives interacting with your product. And so, I think as a product manager, it's like, how do we bring that back? And one of the things, you know, I like to do is telling the story of who should use the product and why, and articulating that core use case. And I think stories really matter. And I think the stories of Bob in Iowa who's on his phone and he's getting up and he's trying to figure out how cold it is this morning, if he needs to put on his earmuffs. Like, so he pulls out his phone and he does blah, blah, blah. Like, that is actually an important use case for your product. And if you can't think of that story and think of what he's gonna do in that moment and think he might actually have gloves on, so it's gonna be harder, so he needs to do something different. I mean, you just don't know all the pieces of every story. of what's gonna make your product useful. But once you actually start telling it and visualizing it, I think our great product managers are great storytellers in the team to keep sort of telling that story over and over again. My arguments with the designer used to never be about, like, you should move the buttons around. It was like, well, look, if, if John comes to this page and he's expecting this stuff to happen, do you think he really sees that it should happen in this order? And if the answer is yes, then I'm gonna trust my designer. And if the answer is, they're like, oh, that's a good point. Maybe I should reorder things or strengthen the bounding for this section. That's a much better way to kind of have the designers and the team really reflect on what we're building, by putting it all in the context of actual humans and users. You know, you need to then advocate it. You need to understand their challenges. You need to talk to as many of them as you can. I found, you know, for me at least, Twitter was a great way to talk to people. And I would go into, I learned how to go into, like, the Justin Bieber crew on Twitter, and, like, which is, like, you You know, young 14-year-old girls who spend most of their day tweeting about Justin Bieber. There's a whole different section of Twitter and sort of hip-hop communities and street communities that use Twitter very, very differently. And you can just learn so much. For me, that was when I was working at Twitter that was great. You know, how you get out in front of your users is always different, but I think it's very important to figure out how to do that. And then, you know, I'm continually listening to feedback. We used to do pretty regular usability tests. We'd just have people coming in and watch them. And, you know, for growth, the big way that we did it a lot was, you know, welcome to, you know, welcome to Twitter. What do you think it is after kind of going through our signup process? You know, and they might have heard about it a little bit in the world, and then they'd go through our signup process. And then if they couldn't articulate it back in the way we wanted them to, we knew we had to do a lot of work to make our process better. So, overall, like, if you're a product manager and you're thinking about, you're helping your team, bringing the context to your company, figuring out how to ship stuff that matters, figuring out how to do the right product to really measure it, and then figuring out who your users are. I mean, it's a, it's a lot of work to do as a, you know, an own-in-one person, and a product manager would be doing a pretty good job. So, you know, the key dos for product, for great product managers are disagree but commit. Really get everybody involved, and then come up with a clear course of action, and then document it, and then get everybody get everybody sort of running well on the plan and keeping the cadence so that the plan actually happens. You know, the don'ts are, don't just try to build what you think is right and say, I'm right, for the team all the time without listening to everybody, having clear hypotheses. Now, I don't mean you shouldn't build what you think is right. Like, you clearly want your team to do it. But you shouldn't sort of be a dictator or a CEO without sort of getting everybody's feedback. Cause the team's not gonna execute blindly. And then, you know, I like to say don't forget where the credit's always due. Because, you know, no matter what, as a product manager, when the product actually ships, there isn't a single line of code or a single mockup that you've actually written. There's a ton of people who've worked incredibly hard to make that happen. And, you know, I always worry that product managers get quoted in the press and get featured on stage sometimes, sort of like, you know, and they, the bad ones take the credit. They're like, ah, I did all this stuff. And I'm like, no, you're just doing that because That's part of your job to help the whole team be successful as to how you are to crisply articulate it in those moments, rather than taking any credit for, for what's actually happened. I find this even worse as an investor now, because, you know, I'm, I'm on the board of a few companies, but, you know, every single thing that they do is 10 times harder than, than, than what I'm doing for them now, too. You know, the product manager's job's never done. You know, as I talked about a lot, once you actually bring the product to market, You'll be working on the next iteration. And it isn't just the set of things we didn't ship, because you actually made trade-offs to ship all those. It's, it's what, what is that data telling us? What, how are we going to make this thing better? What other testing do we need to do now that it's live? Before it was live, maybe you couldn't have been on every device or in every scenario. Now you can go be in all those scenarios. Maybe you can realize this doesn't work in low-light situations or whatever it might be. So how do you figure that out? And look, no products are ever finished. The beauty of what we get to do is take everything, the best thing we can get in our users' hands today, and then figure out what the best thing we can get in our users' hands tomorrow or 6 months from now or a year from now and keep helping your team move forward. No product's ever gonna hit for everybody, but it's an ongoing process of development and iteration to make it better. And that's kind of, great product managers help their team do that. And I think that's, that's super exciting. So.

Thanks.

I'm very happy if there's a few questions that people have to take some after this. So thank you guys again for having me.

All right. Questions in the room?

Yeah.

So you mentioned growth quite a bit in your talk. And it, it feels a little bit like it blends over pretty strongly with product management. I wanted to hear from you is, kind of, where do you see growth? How does it relate to product management? Is it one and the same? Is it a collaboration? Just wanted to see from your experience kind of like where growth and product management collaborate and blend. Yeah. That's a great question. There's been this, like, real shift over the past, call it 8, 10 years, of this, like, new model of, like, growth. And, like, at one level, growth is, just marketing. You know, if you think about a company, you have to do 2 things. You have to build a great product that, that actually works for people and delivers on the core promise. And you have to figure out how to get it into their hands. You have to figure out how to get them aware of it and in their hands and motivated to try it. And when they try it, motivated to get going on the experience so that it actually works. If you think back to, like, consumer packaged goods, these teams were separate. You had a, a team that, that made the perfect toothpaste and a whole different team that, like, made the commercials, created the desire, figured out how to put it in stores, and then you just hoped at some point they, they met in the middle. I think what we've realized in building software is you can actually build as many features and spend as much energy building creative technology solutions to drive awareness, onboarding, motivation, etc., as you can to build the actual product itself. And you can build things that help your really successful users share the product out so your product grows virally. You can figure out how, when somebody shows up, people often, you know, in software, people have always sort of like hand-waved over installation and onboarding. And yet, if you spend as much time building your onboarding experience and your habituation experience to get somebody using your product all the time, you can have them be really addicted. So, I've thought about growth and sort of this new model of growth as really product management for the people who aren't users of your product yet. And core product management is building features for the people who are users of your product, building them for the next things and the, the bigger things you want to augment. And it's just as important to build for the first group that doesn't use your product. And, and so, if you treat it that way, then you kind of blend the core skills of product management and engineering and building the right things for your users and shipping them and all that stuff with the skills of marketing, which are, what does it take to sort of understand awareness and and use data to really understand, you know, how we're putting something out in the world. There's still a room for sort of the skills of marketing applied to all the marketing things that aren't product-based, whether that's events and campaigns and advertising and, and things that, that aren't in your product. But, you know, one example I like to give is, you know, when you used to have a marketing team and a product engineering team, the marketing team would say, hey, we need a landing page for this campaign we're doing. So the engineering team would go, ugh, a landing page. Fine, we'll build you a landing page. And then they come back and say, we need another landing page. And you're like, oh my god, another one? Fine, we'll build you a second landing page. And then it'll be like, a third one. And you'll be like, oh my god, this is the last one you're gonna get. But when you just say, hey, the growth team is ultimately responsible for getting the product in people's hands and everything else, you get the first landing page, you're like, how many are we gonna need? Is this gonna be a thing we're gonna have to do? Then you build a system that lets people who are thinking about campaigns all the time just build and spin up landing pages and make them work really seamlessly. And the growth team has built that. And that's just as important a feature to the company and to your product as building the next whiz-bang settings widget or thing actually in the product. So that's how I've kind of thought about the blend. I don't know if that totally answers your question.

I will grab this for a question. So he asked part of my question, so I'll go to my follow-up. So you mentioned Twitter and growth. And Twitter and Firefox have so many similarities as a product. There is a passionate core group of users who are always vocal and demanding things. Talking about Twitter, like everybody on Twitter is like a product manager. They're always asking Twitter should do this and that. But that also makes it harder to enter for new users. If only experts are using that, how do you guys go about balancing that? Like the core set of users which are very vocal, demanding, whereas the people who are non-users, as you said.

Yeah, look, I think it's a matter of prioritization for the whole company. You know, and, and I, I worry that Twitter hasn't quite prioritized the group I'm gonna talk about, you know, as much as, as much as we were trying to do when I was there. But, like, you have to do both. If you don't move your core product forward every year, other people will faster and better than you. And even your core users won't keep loving your product. And it doesn't mean that everything they're asking for is right, because you can definitely have the very, you know, minority vocal core. That, like, actually demands things that are, like, kooky, and that isn't kind of, like, what's gonna hit mainstream. So, like, you do need, as your core product group, you need to be able to, like, what are the next mainstream things that are really gonna get, you know, the next 50 million, 100 million people to want to care about our product even more and be able to do that. But the growth side is slightly different. The growth side is, of people who are not using my product, what are their objections? And everybody always thinks, well, we don't have feature X. If we just built feature X. That will make them use it. And that's almost never the case. The objections are usually, oh, I didn't know it could do that thing that it already— your product already does. Or, I tried it and it was too hard to use. Or, I couldn't even get it going fast enough and my inertia to actually go and change and adopt your product was too much. You know, for Twitter, we had this phenomenon where everybody was talking about Twitter. Today on Twitter, such-and-such said this. Today on Twitter, you know, you could have learned about the earthquake. Today on Twitter, all these concert tickets were sold. And so everybody was like, oh, I gotta go try Twitter. And they'd be like, they'd go try Twitter and it would be really hard to sign up, very obtuse to figure out, and they wouldn't actually get the hang of it, of what you could do on Twitter. But if you spend hours on Twitter, if I go and spend hours with somebody like, what do you like? Here's how we find the right accounts. What is going on here? Here's what's going on here. And I get somebody to that state, they have an amazing experience on Twitter, but the product itself wasn't there. Especially when somebody had already had something like Facebook or whatever in their lives. Or email or reading the newspaper. Like, whatever they were already doing to discover information, you know, wasn't there. So, the growth team, the way you want to prioritize it is what are the growth efforts needed to really be able to drive that? At Facebook, in the mid-2008, 2009, Tim, over a third of the engineering resources and product resources were on growth. Because Facebook realizes And realized it needed to build all these core engines of growth to keep sustaining the company. Now, that worked so well that they still have a very core growth team. I doubt it's 1/3 now. It's probably more like 1/10. You know. But, but I think like that's, you know, that's the shift in the pull, which is how much should we spend building new core things versus these things that really drive growth.

All right. Any other questions in the room?

Yep.

You talked about growth being a lot of converting new users into users, permanent users. And how do you feel about, like, resurrecting users or increasing the engagement of existing users, where you said that a product manager will focus more on the core product, the core features of the existing users. Is it the product manager's responsibility to increase that engagement and, or resurrect the old ones? Or how do you draw the line, like, further down the line and not just in the onboarding? Yeah. So, so resurrecting old users is the same thing as getting in new users. I try to say non-users instead of new users, although I, I probably mix that much. But, like, You basically have a set of people who habitually use your product. And you want to really understand what that means. And the way we looked at it at Twitter was we figured out how many times you have to use it in one month so that I'm pretty confident you're going to use it that many or more times the next month and the next month and the next month. For us, we found that if you used Twitter on 7 different days in one month, you were over 90% likely to use Twitter on 7 or more days in month 2 and month 3 and month 4. So we said the people who had already gotten to that point had hit a habitual state where Twitter was now part of their lives. And if you aren't at that state where you were using it at least, it was about twice a week, if you were using it at least, you know, approximately twice a week, you weren't at that habituated state on Twitter. And obviously, we wanted you to be using it every day, but I found that, you know, the at least twice a week was a good enough proxy in those early days. And so, for a product like yours, it's who are the people who've gotten to that sort of nirvana state where it's now the default browser? You know, and you guys might look at different things. It could be, are you the default browser on the computer? Are you launched every day? Are you, you know, launched 3 times a day? You know, you don't necessarily want somebody who uses Chrome for 90% of your things but keeps opening Firefox for like 4 websites. It's not terrible, but that's not nearly as good as when you're doing 90+% of your browsing or 100% of your browsing in Mozilla, right? So you want to figure out what that core user bucket is. And what that habituated thing is. And everybody who's not that, the growth teams, in my opinion, the best growth teams are the ones who are like, how do we get people into that state? What features do we need to build to get them into that state? And, and most of those features are usually awareness, you know, retention marketing, onboarding types of things. But sometimes those features are viral and social features where we say, how do we take the people who are really sticky Build more things for them so that they pull more of their friends in. And by pulling their friends in, that pulls them on this loop where they end up getting into that habituated state too. So like, you can do both, sort of. But that's how I think the growth team is like, how do I get you into the habituated state? And then once I have you in the habituated state, how do I get you to do more? Or, as world trends are shifting, video is now a big deal. Nobody was watching video before. Everybody's watching video now. We need to make sure that we don't lose them from that habituated state because they're gonna spend more time on video. So you go and do work on that, for example. Does that answer your question?

Great.

All right. So we have a question from IRC. Do you have any experience or advice for working as a product manager for a distributed team?

Oh. Product manager for a distributed team. So, I have never done it. So I will, my advice will be much weaker than all of you who work in an incredible distributed company. I think the The 2 things I would say are the communication needs to be extra clear when you're in a distributed organization. I think the, the clarity of, of whatever gathering you've done and what decisions were made just need to be extra clear in the written form. Because, you know, one of the nice things when you do work proximately is even if there is some, you know, disagreement or confusion around a topic, you can just go over to somebody, usually with like face-to-face looking into somebody's eyes and Back and forth, you can kind of get to a good conclusion. It's much harder to do in a distributed world. But even then, getting on the phone, getting on a Skype, getting on a, you know, any face-to-face, you know, is really important to work out any disagreements or confusion around something. I think the second thing that I would, I would emphasize further is making those regular communication things where everybody is live together even more regular. And more consistent, so that when any issues crop up on the team, you know you have the forum to air them. Like, one of the benefits of having a great weekly meeting where everything gets aired is when things come up in between the weekly meetings, you're like, I'll just keep this on the, I'll make sure that this needs to get raised in the weekly meeting, and it will get raised. And you have that perfect cadence to do that, as opposed to every flare-up becoming an issue and not being able to find people to deal with it. So I think it just— creates even more importance on both the written communication and the live cadence of communication.

Cool. So, do you have any thoughts on building something for early adopters first and counting on them to set the trend for larger groups of users?

That's a great question about early adopters. Look, the way I like to think about building something is, is not, is, finding your first passionate community. So whether that is tech early adopters or other things, you are much more successful in a product if you can build it and you have 100 people using your product religiously and changing their lives around your product than 10,000 people who touch your product lightly at the very beginning. So almost everything you build, you want to find that, that small group that's really passionately converted. Once you do that, Then, then the gut check is, is this replicable? Is it only interesting to them because they have a very unique set of needs and skills and actually the rest of the world won't ever care? Or can you actually go and sort of grow it out from there? And that's a little bit of instinct. It's a little bit of smarts. It's a little bit of being realistic about who those, those early people you are targeting, you know, are actually the right ones. But you're much better off kind of finding that. And, and by the way, as an investor now, that's exactly what we look for too.

Yeah.

When we're trying to find the next companies, you know, everybody's like, look, I have like this many users and this much things. And we're like, okay, wash all that away. Do you have a group of people who are actually like changing their lives because of your product in the really early days? And if you do, it has a chance to get much bigger. You saw Facebook grew in colleges in that way. Snapchat grew in high schools. And like, what was exciting about Snapchat in the early days was a whole high school was super passionate and they were like, that's all we communicate through now. Like, that's a, that's an amazing, like, green flag for, you know, for an investor. And so you want to think about the same thing with every product you build.

So, go ahead. Oh, we got another question on IRC. I can't see it.

OK.

OK. So this question is, would like to have more examples of how the really good note-taking and all the extra time after meetings paid off. So, How did this save time in the long run or prevent big problems later on?

It's a great question. I'm not sure, you know. You know, I think, I think one, a couple of the key ways we did is we communicate our notes very broadly in the company. So there were a lot of times where I would get questions from people like in support or from my CEO at Twitter or other things where they would ask lots of questions about the notes that we had and I could kind of answer those. And, we would actually, that would surface any, like, real discussions that needed to happen much, much sooner. So if I hadn't written it down and sent it out, I never would have gotten them thinking about it early. There was one feature, the feature I was talking about that shipped after I left, but the one about, we want to do this personalization based on all your website browsing history. We were talking about that for a long time before legal sort of responded one day and jumped in and said, hey, we'd love to talk to you about this. And it opened up a great legal conversation. very, very early. And a team that didn't distribute things as publicly and as, you know, concisely and as clearly communicated never would have triggered other people to think about what we were working on and give us ideas or whether it affected them. So that was, that was one. Within the team, you know, I'm, I'm not totally sure, honestly, if it makes any difference of people walking out of a meeting. Cause I can't remember all those situations where, you know, everybody might have walked out on different terms. But I certainly know that, you know, once it's written down, you know, and that becomes your action item, and it was much easier to say, hey, did you get this thing done? You know, people can't be like, oh, I didn't know I was supposed to get it done. 'Cause it was written right there.

Cool. I think this is gonna be our last question. From IRC. How do you prioritize through simultaneously occurring fire drills? Or, you know, Do you do scorecarding? How do you choose to focus on one fire over another?

That's a great question. The reality is, we have this great myth that we can, like, parallel process as a team, and that parallel process as a company, and we can do more than one thing at a time. And, like, oh, our team can get, like, all 4 of these done this quarter. And, and the reality is, we're all humans, and actually a lot of things do chain up, and at some point, the CEO or the PR person or the chief Council can only focus on one thing at a time, and everybody has to know how to prioritize. So the trick is you, you have to pick. And you literally, I, I like to think about writing everything down. I call it a one list, which is like, if you just rank everything that was important in the company right now and stack rank them and legitimately ask yourself, is number 7 more important or less important than number 8? Cause if it's less important, move number 8 above number 7. And if not, it is more important. And then you can use that to sort of Track it, and then when somebody comes in with a fire drill, you're like, okay, which of these things are you more important? Because no matter what, in terms of paying attention to your new thing on the fire drill, you're gonna bump something else down. And I think as long as people are honest about that and not they're like, oh, I can have my cake and eat it too, it works. And then, and then you fundamentally have to make the call. Like, part of what, you know, an organization of trust is, is somebody has to make the call that says, I'm gonna focus on this over this. And if the tie has to get broken, you can move it up in the organization.

All right. So that concludes our March speaker event. We've got a— oh, there's one— one more question in the room. Javi.

I was hoping you could just comment a little on when developers are your early adopter first audience And I know Facebook Connect and Twitter have kind of different stories there. So if you could share some wisdom, that'd be awesome. Thanks.

Oh, man.

Building platforms is a whole different conversation. I actually put up a fun talk about building platforms on my Medium page, too. It's a slide deck kind of like this. We have a team now that takes my crappy slide decks that I make and makes them look all beautiful like this. But I made one about platforms. When you're building for platforms, it's a totally different value exchange, and both sides have to understand the value exchange and the risks that they're taking on. I actually think Twitter actually is a great developer platform. There's multiple, there's several multi-billion-dollar companies built on top of Twitter that people don't realize. Companies like Hootsuite, companies like DataMiner that were all enabled because of Twitter. And that Twitter sort of didn't actually kill that many things except for a few other clients that actually was somewhat in the way of Twitter's model. So I kind of reject the sort of like, Facebook got it right and Twitter got it wrong. And if, if you'd asked me in 2008, everybody hated Facebook's platform because we were changing the way that Facebook worked, and everybody was like, you're killing all these developers. So, that's sort of a really interesting topic in itself. Look, when you're building for developers, it's the same thing as building for users. It's, today, what can I give you that makes your value better? And with developers, it's, what value are you giving back to To me and my platform. So it's, how is this a good mutual win-win type of thing where collectively we're making the experience better for our users rather than just me as my own developer making the experience better for the users? And if you have developers on board with you where they're making it better and you're opening things up that make it better for them, that's great. And some of them will end up building their own opportunities really big, and some of them will end up plugging your holes that you'll eventually plug.

So, Awesome.

Yeah, it's on Medium. Thanks, Josh.

Thanks, Jesse. We have a mini backpack full of swag here.

Perfect. I like these.

So our next event is April 27th in San Francisco, 10:00 AM Pacific. Thanks, everybody. And thanks a lot, Josh.

Thank you.

Oh cool.

Yeah, yeah, it's been a random one.