Managing Change & Building Winning Teams with Roblox CTO | GLOW24

Well, hey, uh, so everyone excited to be here? Reminder, I'm Andrew Lau, co-founder and CEO here of Jellyfish, and I have the pleasure of talking to Dan Sturman, CTO of Roblox. Most of you know what Roblox actually is. I mean, if you have kids, you definitely do. I've got ones and I'm definitely familiar with Roblox, but I'm excited to talk to Dan because I want to learn about how he got here, all the things he does. You know, the man's a legend here. But before I go into that, Dan, do you mind explaining to the audience here for folks that may not have touched Roblox, like, what is Roblox and why should they know it?

Right, right. So Roblox is first, it's a creator platform. We don't build any content. And it's a creator platform generally for 3D social games. So you can imagine, not like Candy Crush, that's, that's what's meant by a 2D game, but you're moving around a world, um, that is created by some third party who's built on the platform, and, um, you're doing whatever. I mean, the, the experiences on Roblox go beyond any one profile. There, there can be strategy games, there are just kind of hangouts There are music concerts, there are shopping experiences, but the common theme is it's 3D and you're doing it with other folks. Then we have a creator community which is a few million strong who are building all sorts of content. Our job is to put the right content in front of the right users. We run the whole backend economy, which is, I think someone told me, slightly bigger than the economy of Belize. And we enable all the 3D engine to work. We ensure our users stay safe and things stay civil. And we run it kind of in some sense where the, the, the biggest cloud provider you never heard of, because we're running all the compute. No one's paying for any compute, storage, whatever. That's just all kind of part of the package. Got it. So that's what Roblox is.

Thank you. And, and, you know, we probably got, um many technologists here on the horn. And so now we kind of understand what the business is. Can you help folks talk about the scale in terms of, I don't know, how should we talk about as technologists? How do you describe scale? And then also from a people perspective, how should we think about this?

So we have about 70 million daily actives. So that just gives you some scale, right? There's people across the globe on the platform all the time. We no longer really have a time when nothing's going on. When I started, there were times where very low. Now it's, it's kind of on all the time. We run across, uh, about 23 data centers worldwide, so we're, we're pretty broad. We run our own data centers, by the way, um, so that's kind of a difference from a bunch of other businesses. Um, peak concurrence for us will be in the millions of users on at a given moment, like at a big event or something like that. Um, so million of concurrent users interacting in 3D kind of across the globe. It's the way to think about the scale.

Awesome. And how big is your team, or how big is the— how should we think about the company as a whole and the technologist side of things?

Yeah, so one interesting thing about Roblox, one thing I love about being CTO here is this is through and through a technology company. We don't have a sales team. Our marketing team is small. Our finance and HR teams are what you need them to be. The biggest function here are product and engineering, and engineering's about 1,600 of the 2,500-person company. And, you know, product and data science and so on is a few more hundred. And then the rest is kind of recruiting is probably the next biggest arm and you know who they're helping out. And then we have, you know, a modest-sized kind of corporate G&A sort of team across marketing, finance, HR, all that sort of stuff.

Love it. Well, thank you for framing it. And I think like sometimes, you know, we look at things on the internet and it's hard to intuit the scale of things because there's the way there's how we see it, and then there's actually how the world actually—

That's right.

If you do your job right, we don't know, right? Like all the stuff that's actually happening behind the scenes. So, which actually I think leads me to actually, I think what I think a lot of folks want to hear about, but what I want to hear about. So you and I have actually known each other, gee, for almost 7, 8 years now, maybe. Yeah. And I think, you know, we first met when you're doing kind of large-scale data processing at Cloudera, like a really specific kind of technical challenge. I think even then we had a lot to chat back then about the IBM days, just to kind of roll the time back. But actually, you know, as people kind of go through their own careers, I mean, you've had a very, very, very successful one. Not just do we lose the intuition of like what Roblox is behind the scenes, very little intuition of like, what's your job like anymore? Like you are a, you know, very experienced trained technologist. I think some people think like, okay, you're sitting there writing the code, right? Like you're building the animation model on this thing, right? Or like, I don't even know, like, actually, I don't even intuit, like, with 1,600 people, like, how do you spend your day? Like, what's going on?

Sometimes I don't know. So that's a very good question. The day sometimes just happens to me. But, you know, a few things I'd say, particularly Roblox, one of the things I love here, I'm, you know, different jobs I've had, sometimes you're more, you have bigger what I'll call business or people challenges, others you have more technical challenges. Roblox, skews pretty hard to the technical because we're such a technical company. So much of the time across the executive team is spent on product strategy and how we're going to build it and how quickly we can build it. So I do get a fair amount of technical details. Now, am I writing code? I would hope not. Like, if I'm writing— if I'm pushing production code, we have a problem. That is not a good idea. I still do write code fairly often, but it's generally hobby projects or things like utilities I'm building in the company, not Nothing that's going to production, so everyone's safe that way. Yeah.

If I were to look at your calendar, is it just like a grid of meetings or is it flights all the time or like, okay.

That is right. They range from everything from— we work a lot by review here. That's the way we manage. For example, every quarter we review all the top-level organizations and their technical strategy and accomplishments for the past quarter. It's a pretty intense cadence. Where those, we call them groups, they're about, you know, ranging from 200 to 400 people in each of them, kind of show up, tell us what they've gotten done, but then also tell them how what they've learned is changing the strategy they've laid out and what their strategy looks like now over the next 2 years, right? So kind of always iterating on the 2-year outlook on that strategy. That's an intense number of weeks, once a month. But in general, doing a lot of what we call office hours here, which are not what you think of as office hours, but they're basically ad hoc reviews on any given technical topic. Of course, with 1,600 people, there's a fair amount of time spent on both implementing but designing the HR processes here. As a rapidly scaling company, we've had to do a lot of that. We've done some things a little bit differently in that regard. And then, of course, the usual one-on-one sync with people and so on. You know, honestly, those are the means I try to keep to a minimum. It's not that I don't like meeting with folks, but I just find we scale better when we are in meetings with a goal and a purpose and are trying to accomplish it.

And an agenda, right? Like to run through.

Yeah, that's right.

I'm going through that myself right now. I'm like, calendar is gritted out and I think trying to bring more structure to it is actually a good thing.

Yeah.

How do you, so you describe meetings that are kind of more like quarterly strategic reviews and I think that's an important rhythm, but do you have like a, how much of your life is tactical? Do you have like daily reviews, standups or those things or like, and then Or is your world entirely strategic now?

No, I mean, I will have daily, but not on a regular basis in the sense that if a project is in trouble or we need to really shift the strategy quickly, then we will for some amount of time do, you know, a multiple times a week standup check-in sort of thing. But I don't think it's success at this level if you're doing that all the time every day, right? Like that's, right? And it's not like even myself and the VPs here who are my direct reports, you know, there's 8 of them. So if we're all getting together, like they should all just be working on their stuff. At that level, they should be driving with their teams. So I try to avoid anything that's a daily cadence except as a tool for a temporary daily cadence when necessary.

Got it. All right, so I heard from you, it's an exception, not a rule, but it's part of your toolkit. When things are a little sideways to actually deploy it.

That's right. And Bobby, not always sideways. It may just be there's a huge opportunity we just kind of ran into and we want to move fast, right? And part of the value I find in that sort of tight sync isn't to micromanage the team, but to help get obstacles out of their way as quickly as possible and avoid a loop where they're waiting to get time to come back and show you something or something like that. You can just iterate really fast.

You're driving focus, right? You're clearing the way, making sure everybody understands this is what we're actually doing. But also while you're doing it, you're clearing out things from them on that.

That's right.

Got it. Well, I think you're leading into this and I think, you know, with 1,600 people, like, how do you organize? Like, what's the shape of this whole thing?

Yeah. So it's interesting. We, like every organization, we struggle with these— should we organize functionally or topically, right? Like you see every business do this. They reorg between like a functional organization, then they have GMs or back. The way we structure here, we don't really have GMs per se because that there's, there's really only one business for us at this point, like only one revenue stream, right? Um, but we organize in a set of groups and groups are large thematic strategic areas. So we have an infrastructure group, we have a trust and safety group that worries about keeping people safe. We have a group for the engine that does all the graphics on the platform and so on. So like we have about 8 of these groups and then they're sub-organized into teams. Um, you could think of a team in infrastructure as like the storage team. It's kind of obvious. But like in trust and safety, like account integrity would be one team within our trust and safety organization, right? So again, you can see these, these groups and teams are organized thematically based on a strategic goal. The engineering org structure mirrors that structure, but then there's also a mirroring product structure and a data science structure and so on. And then those teams and groups are actually the management units in the company, typically run jointly by the product and the engineering lead. Within that group or team, right? They tend to be primary here 'cause you can't, you can't do everything by committee, right?

So what I, I'll just restate it back to you. You, you kind of break apart into kind of functional areas, but then in those things they're organized, in those things functionally by role. And then like, and so kind of like on a, I don't know the better word for, but a pod structure by function and that stuff.

That, that, that's exactly right. And, and that's how it works. And we build tools to help our, so those reviews I mentioned, they're not engineering represents and then product presents. Presents, and then data science presents, or so on, the infrastructure team group will stand up and present, the economy group will stand up and present, jointly between all the folks in that organization across, kind of matrixed across the organization, they own that deliverable because a strategy comes from the thematic, right? So that's kind of how it works. Now there's of course things we do on an engineering-wide basis and so on, like, you know, Sorry, someone just tried to steal my room there. Get out of this dance room! So, you know, some of the work we've worked on, like, you know, how do you understand the progress you're making, the speed you're making? We're currently going through a Jira restructuring, you know, like that doesn't need to involve the whole company, doesn't get discussed in those quarterly reviews, but I'll get a sync on those, you know, once a month or something like that.

Got it.

Love it.

That's the cross-functional fabric that kind of runs the business as you drive your objectives.

I heard you touch on it briefly, around reorg.

You said it very fast, it flew by, and actually it made me think of something. I was in a— I was in a— I watched a panel discussion recently where people were talking about this discussion of functional versus kind of GM model and how reorging is actually a feature, not a bug. But I don't know, like, is that a hot take? Like, how do you process that?

Yeah, I personally find reorg as a tool is overused, and sometimes it's overused because senior leaders, it's one of the like tools in their toolkit. And so they kind of reach for it and it's kind of a tool that's available. But my view is no org structure eliminates problems. You have a conservation of problems. You just get to pick which problems you want to have today, right? Because there is no— when you're dealing with large organizations, what's the problem? The problem is everyone can't talk to each other all the time. That's really what you're solving for. And then you're kind of deciding which people are going to talk to each other more. And have their goals aligned and kind of have some structure reporting through the business. This is why you don't worry about reorgs in a 10-person startup. I mean, there's nothing to reorg. Like everyone can talk to each other all the time, right? So I'm always very conservative with reorgs and generally I will do them when already folks are functioning in a new way and you've done the hard work of getting people functioning the way they want and then you say, okay, well now it's silly to have the org structure we have. Let's just get it to recognize what's already going on in the organization as opposed to a way of leading. I don't tend to, I tend, that tends cause more confusion. You have all this time when people are trying to figure out, well, we've been reorged, but how are we supposed to operate now? We're not quite sure. And they're trying to figure all that out. So I tend to lead with the doing, lead with the let's operate the new way, and pretty much always find you can get folks to do it. If you have leaders who aren't able to do that, I think that's a good signal that maybe they're not the leaders you need. Like if they are too focused on who is my boss and who are my subordinates, I tend to not like to—

You're missing the spirit and the goal that they're actually—

That's right. Yeah, yeah, we're all one company. It's very much a first team principle here at Roblox that, you know, we're all working for Roblox and everything else is secondary.

Got it. Well, I'm just still a tip I heard from Dan, at least from Dan's words, like reorgs are a coarse hammer. I know you didn't say that, but I'm actually reflecting back on that and lead with what you actually want the shift to happen for. First and that the reorg is maybe a reflection after, like once you've actually gotten there. But reorg is not the goal in itself. It's a recognition of what needs to be happening, not the motivator.

That's right.

Got it. Well, hey, thank you for that. Well, speaking of which, because you like, you talk about how you actually engage with the team and really that who's talking to who. You've got 1,600 people. How the heck do you actually talk to folks? Like, I don't know, like, you know, we've got a couple hundred folks here and like we still try to find our fabric, but like, How do you engage with 1,600 people? Like, how do you make it work for you and them? Like, how do you balance it?

It is, it is difficult. Um, you know, like I said, those group reviews are good because a lot of— not everyone comes into those, but a lot. You at least get to see the leadership. From my point of view, I tend to see, you know, at least 2 levels down in the structure, if not 3, in those reviews. So that helps.

I can't do the math quick enough here.

So like, in other words, how deep does the org go?

And like, maybe it goes—

we go about, we go about 3 to 4 levels, maybe 5 in some cases, right? So in other words, there's my directs, of course, in those meetings. Um, there's their directs, which would be a director generally at Roblox. Um, and then I tend to also— sometimes the managers get up and present something on a particular focus area or something like that, right? And by the way, when I say that, there's product management peers, there's their individual contributor, senior engineer peers that it looks. I always keep an open door policy and actually a non-trivial amount of my week. I mean, it's not a lot, but I would say maybe in a typical week, somewhere between 1 to 2.5 hours are, you know, ad hoc one-on-ones. Someone wants to meet about something, it can be any topic. And I try very hard to keep an open door and make it possible for anyone to come talk to me, even a, like a summer intern. So I do that. Definitely. And this is the hardest part for me, you know, taking advantage of company social events, walking around, meeting people, talking to them. You never know what you're going to hear. I mean, we may talk about this later and return to work in the office. One of the big drivers for us on that, my first week back in the office when not everyone was back, but people were just starting to show up because we could, right? I thought, oh, this is going to be great. I'm going to pick up one thing a week probably that I would have missed totally. And I found myself picking up one thing a day. You know, that just wouldn't have happened otherwise. I went, wow, that was just like a mind blow on that. But you kind of go around, you meet people in the kitchens, you— this is why I'm a big advocate of having lunch on site. It's an expense, but means like there's people to see, sit down with teams, you talk to them.

So, you know, maybe we're blurring into this, like, this is like, let's just jump into this part of it, right? Because we're talking, I think it's entangled. Of course, now I think about it, it's entangled. You said 1,600 people. Maybe orient us. Where is everybody? And then, like, and then you're already talking about, like, you know, we're talking about how do you engage with them. But I think this whole discussion of, like, hybrid, because, like, gee, I think back the first time we met, I actually found you at your office. I sat down at your desk with you, um, and we were looking at— I think we're looking at your computer together on that thing, right? Um, that was 8 years ago, and the world's different now, right? Like, um, but So, so maybe help orient me and everyone, the 600 folks, where are they? You're describing a new fabric around lunches and social events, but like, and I guess like leveling up, like this is the change in work from hybrid to in-person. Like, how are you engaging with that?

That's right. And it's been, I think, I think that's been probably the biggest management challenge of, let's say, the past 2 years for anyone, right, is trying to figure this out. And I don't think we're done figuring it out yet. I think COVID changed not just industry, but like, I have kids in college, it completely changed the way education is delivered. I think it's changed to some degree the way people kind of socialize a little bit now, right? Like, before, if you said, hey, I'm going to jump on a video call with some friends, no, you just call them on a phone. But now people like do video conferencing a lot more just between friends, right? Yeah. So I think everything has changed in that regard. But that said, we found when we came back, we're a company that very much needs innovation to move forward. We're a very tech product-oriented company. We're not going to grow because of, you know, just better sales coverage. There's like other tools different businesses have. Ours is about innovating the product and knowing where we need to go and where our users need to be and doing that as quickly as possible. And what we found is in-person time was critical to doing that. Essential, like existential, I would almost say. Okay. Um, so right now we went through a tough thing. Um, it started last summer, but it kind of concluded— it's in the process of concluding now. We told the majority of our employees who had, during COVID dispersed everywhere— a lot of them had— we're going to be in San Mateo unless— I mean, there are exceptions to that. We, we have an office in Shenzhen, China, for our Chinese partners. Our data center workers obviously need to be in the data center they're in. That would be silly to expect them to be in headquarters. But by and large, we expect the majority of our employees to be here. And for the ones who aren't, and it's a very small number, and it kind of mirrors the number we had who were remote before COVID high expectations on travel back to headquarters on a regular cadence to stay in the loop. And in fact, this week we're doing something we call kind of quarterly Roblox week. Which we do once, once a year, where pretty much everyone is on campus all 5 days this week. And there's a whole bunch of events. The groups are syncing with each other. There's folks kind of digging in on new strategy directions and so on. It's kind of week-long festivity work session sort of thing. But just getting that time back together was important. So we settled on a 3-day work schedule. The same 3 days was important. You know, some folks are saying 3 days, but pick your 3. The whole point was to get people together.

Yeah, it doesn't work if you're not—

if you're not— that's right.

Do it like, and they're doing different days. It doesn't make sense. That's right.

And particularly when you're saying, what we're looking for is the accidental conversations. That's what we thought was missing. It's the accidental and then to some degree also the more heart-to-heart conversations. We found hard conversations over Zoom just weren't happening, whether it was a manager giving feedback to an employee or a technical point that some engineers just couldn't agree on. Everyone's just being too frickin' polite over—

Yeah, I agree with you. Arguments are the thing that's missing, right? It's not that no one speaks arguments. But there is a ritual about having an argument and then getting to resolution. Right. And the hard part with the remote part of it is there's so much friction to finish it. Right. Like there's a physicality sometimes around finishing.

Right.

If I have to pick up the phone and call you or initiate a new Zoom, sometimes things don't get put to bed. Right.

Right. Yeah. That in a sense was, I think, the really tricky observation because obviously telling folks to do this was very controversial. We lost a bunch of employees over it and so on. But if you think about it, folks were making a good argument. I feel more productive when I work from home. Right? I don't think they were wrong, but I think the key observation was the output of an organization wasn't necessarily the sum of the output of all the individuals because they had to be pulling in the same direction. It's like in a game of tug of war, if people are all pulling at different times, you don't get the— you don't get the— emphasis of everyone pulling at once in the same direction. And that's so essential. So I think that was the dichotomy. Employees were rightfully saying, hey, I'm super productive. Management, what the heck's your problem? And we're going, but we're not getting stuff done fast. We're making misses. And I think a lot of tech companies in particular have come to that realization. And it's been a tough realization, right? So we do do only 3 days in the office. We felt that was enough. It gives folks 2 days to you know, not have the commute and so on, which I think is a pretty big burden off. I love it too. You can just kind of get to what you're doing better. You can deal with home stuff a little bit if you got to multitask that way. But the 3 days in the office means there's like core time to sync, connect, and to run into each other. Got it.

Well, hey, I think there's a lot of good tips there. And I think like many organizations are going through this transition. So like, thank you folks for blazing the trail there. And actually, helping us think through some of those things. Because I heard a few tools though too, which is an emphasis actually around that interplay, around that, for that serendipity in those conversations, and sometimes working through the arguments. You guys do it over a 3-day window that has to be the same 3 days, but then toolkits like lunches and social activities as encouragement to actually have those dialogues. Or bits of that, right? I heard from you.

Yeah, to some degree, if you're gonna make people come back into the office, you got to make sure this other stuff is going. You can't just take it for granted. It's got to be happening, or What's the point? You just kind of wasted everybody's time, right?

I mean, the underlying theme here is around intentionality that I hear, right? Like that it's not— we can't just fall into this.

That's right.

I mean, look, the last 4 years have been crazy. There's no way about it, right? We've been in a world of change.

Absolutely. Absolutely crazy.

Speaking of change, I don't think we can actually have an interview these days without asking the question. Lots of things are changing. So GenAI is in play. Like, how are you folks changing? Like, I mean, I, I know like it almost sounds trite now to say this, but I think it's affecting everyone. How are you— how's Roblox and how are you engaging with that change?

Yeah, no, I think that's an interesting question. Well, we started engaging pretty early. Um, I saw generative AI as a huge, um, opportunity for Roblox and also a huge threat if we didn't deal with it. Why? Because we're a creator platform and What GenAI really symbolized to me, I mean, it's gonna do a lot of things, but it really is the first real productivity enhancer for creatives that we've kind of come up with at scale. I view it as taking a farmer who's had a brace of oxen pulling their plow and moving to like a John Deere, right? It really does accentuate and accelerate the work they're gonna do. We're entirely a creator platform, so we had to think, Well, what happens if, you know, someone else does this? That's a real threat. If we can do it and we could drop the time like by a tenth, by a hundredth, what it takes to create an experience on Roblox, think how much better our platform will be, right? But there's other things also more subtle. So for example, we launched voice communication on the platform I guess now about a year ago. And the biggest challenge in launching voice, it wasn't dealing with the scale, it wasn't getting voice streams mixed properly. I mean, spatial voice, we had to deal with the physics of it and all that. The hardest part of voice was making sure it was safe. We have a lot of kids on the platform, and now voice was already limited to 13+, but we got to keep them safe. Doing real-time, like real-time, how are we going to real, in real time, discover if there's bad behavior on the platform? That's crazy. And now this—

You can't afford millions of moderators running around doing it.

It wouldn't scale and they wouldn't be fast enough even if you could. And they wouldn't do a good enough job. The problem with a lot of moderators is consistency now becomes a big challenge. So we built using large models a real-time voice toxicity detector. We use the large models here. It was interesting to really train a smaller, more compact one, but it was the way we could start to build. There was no training set per se that you could use, so we really had to leverage some of these new AI techniques to make this happen. It's turned out great. English toxicity is way down. We've launched now in Spanish, French, and I think Portuguese. We have a lot of users in Brazil, so that's why Portuguese.

Brazilian folks out there, that's my bit of Portuguese. I know.

So I mean, it's just an example of the sorts of things this is opening up, but it's also opening up now if you're in Roblox Studio. We have Roblox Studios, our creator tool. If you're coding, we have a code assist, um, that's specialized for Lua, which is the language that you code on in Roblox. Um, we have a general assistant and you can say, hey, can you kind of "Build me a scene that looks like this," or "Adjust the following things." So you can talk text to it rather than having to know where all the parameters and the pieces are. You could search the marketplace for just the right piece you want to pull into your experience. So it's already accelerating creators. And I expect long-term where we're going to end up— this is the challenge you have the team. If you took the first chapter of The Hobbit and you drop that text into Roblox, do you get the Shire? Like, that's, that's what I want them to be able to do, right? And or any book. I just picked that as one example. It's one we kind of fixated on here, but it could be any, any descriptive text. You should be able to turn that into an experience. And there's so much good descriptive out there, right? It's just a great way for humans to communicate what they're looking at before you get into 3D. So that's what, that's what we're chasing after. It's going to be a few years to do that, but I think we're going to get there.

Got it. And then, so these are all tools, um, and how about internally? Are you—

how are—

how's the company engaging with your 1600 folks, how are they using the GenAI world? Yeah, that might be a trite answer, but I'm curious, have you guys made a decision or attack to that?

I think there's a few things as we think about generative AI that impacted the way we operate in engineering. I mean, of course you have tools like Copilot and other code assists that we're starting to use aggressively and we find— I don't know where people think this is going to replace coders. Like, I have used these tools, like they're not that good, but they absolutely accelerate you. Again, it's a really good job complete, right?

Like, I'm being a little ingest there, but it's that for me.

For me, it's also you don't have to do the normal flow. Wait, let me look at the docs for this API. Let me go back to my code. You could just ask the system to sketch it all out for you and you can go from there. You don't have to do that loop. But I think there's bigger things. One, you got to watch costs. Everything we've done with generative AI, we've done while very carefully watching the price tag. Nothing gets launched at Roblox as just a large language model. Everything is distilled, everything is quantized, everything is done in a budget. And understood what the TCO is and what the trade-off for the company is. Right. So a lot of companies have kind of exploded their infrastructure spend generally on a public cloud provider with this stuff. We've been very, very careful on how we manage that. We've had to put a whole bunch of structures in place to make sure we're both efficient on that spend and not slowing folks down.

By the way, I mean, I'm catching a theme about Dan and how Roblox executes. It's a, it's a deliberate intentionality about everything you guys are doing. So You're marking a spot, a thing that we got to do, and you're doing it in a way that is not like free-for-all. We're doing it. We're going to be deliberate and execute to that part of it.

Yeah. In fact, one of the company values, probably the one quoted the most, is take the long view. And that's exactly that. Like, how is this going to play out? Just knowing what you're going to do to get the quick win is no good. And with generative AI, we had a lot of teams that wanted to do that. Oh, if I just do this, oh, I can launch this cool thing so fast. Yeah. Okay, but what happens later? Yeah. No answer, right? So like, that's where you got to go and rework your plans. And really understand how this all plays out. Kind of along those lines, understanding with voice, for example, that it was moderation and safety that was actually the longest pole in the tent.

Yeah.

Not all the stuff you think about if you're going to sit down as an engineer and write, you know, a voice system, right? So, so we've had to think about cost, but we've also started to build— we've built a pretty sophisticated internal ML platform that does tie into large models that are mostly used for synthetic data and assist with the training process, a little bit of prototyping, but that's about it. But can make it easy for folks to get to these resources and use them and understand what's going on. We've been very careful about our datasets. We thought very long and hard about which datasets are okay to use for these things, which are not okay for these things. Our creators have to say it's okay before we use their data. That's key for our relationship with the community. Putting all that in place was a fair amount of work as well. It has changed in large part how we operate because every team pretty much is starting to touch this technology or is impacted by this technology one way or another.

Yeah. By the way, just for other folks who have been saying it for myself, I love how you're actually attacking this in multiple layers. There is GenAI that and that changes your product. Like, how do you leverage this to do something you couldn't do before, like moderation, right? So that is a clear business value. You're actually connecting to the, you know, the business outcome. You guys are also interesting because you guys are a dev tool provider yourself. You're a platform. And so you're actually saying, how does GenAI help my constituents, my customers, my users in that way? That is one layer of it. And then you're talking about the intentionality of actually how We work internally around this stuff in a thoughtful way. And all three of those are done with intentionality. So often, you know, I talk to folks and they're asking about this and I actually think it's a great framework around like, how does it influence your products? How does it touch your customers in that way? And then actually, how do you attack your team? How does it influence your own team? And then all of those things need to be done in concert with intentionality about that.

That's right. Or you end up with a big gap in your plan is what it comes down to. And now you're picking up the pieces. Is real, right? I mean, I'm not going to name any companies, but you look at the financial reports of some public companies and you see where they're in tech and you see where like their bills are going on this stuff. I mean, I, I don't know how I would answer to our shareholders or our users like if we started exploding stuff like that. It doesn't work.

No, it's in— and it's a, um, and also you're engaging with it without— it's an opportunity, not fear, right? I think there's both of them. Yeah, I think sometimes people can run one way too far, right, on that stuff. And there's a balance about that.

I've definitely found when you focus folks on what does the whole solution look like, they actually build it better. You get a better solution in the end, not just one that also fits some more requirements. Like, getting folks to focus on how do you make your model as small as possible and still get the job done— yeah, distill it down— leads to a faster model, leads to better latencies, leads to a better product experience.

Less money being spent, right?

And that as well, right? But even if the money was the kind of stated goal, it forces an element of craftsmanship in building the solution that overall makes things go better.

Yep. Refined, refined detail, not coarse hammer. Well, I actually think we're touching on this because I think when we look at all this stuff out there, the world's changing fast. And I would say, in your view, what are the biggest challenges facing like engineering leaders, like what you do, like technology leaders, like product leaders, like, and like, and teams today, like, how should we think about that? Because a lot's happening. Like, what's the big thing that we should be thinking about? What's the, what's the, what's the challenge here?

Yeah, so I mean, we just discussed Gen AI from a technology framework point of view. I think that's the biggest one. I don't think we need to rehash it, but obviously that's, I think, I can't think of a business that's not going to impact. That's even not a tech business. I mean, I imagine No matter what you're in, this is going to change the way you operate because of the tools it's going to give you to do what you do. You know, there's also, let's be honest, like there's the financial landscape depending where you are as a company. If you're a startup, it's trickier to get funding these days. If you're a public company, investors have moved to wanting to see more in the way of profitability returns than than just growth, right? So it's changing the way we operate and kind of as a community, we're all kind of running in one direction, right? And we're like, whoa, whoa, whoa, the rules just changed. We gotta switch a different way to some degree. And that sort of dynamic, that's, it's gonna impact everyone. Even if you're not sitting in the C-suite and not dealing with day-to-day basis, your bosses are gonna make decisions that, you know, you're gonna go, wait, what's going on? I think it's important to understand that context, right? Of what's going on. The talent in the industry is kind of, The talent situation has gotten a little bit better recently. I don't think that's for the long term, but we've been able to pick up some great talent over the past few years. And part of it's been our intentionality on how we recruit, but lately it's just like the competition all went silent, right? So we just get to pick who we want, which is great from building the sort of team you want. It's a good time to go make those key hires that maybe we were struggling with before, right?

If you had to pick one, Like, where would you go? I mean, there's, I mean, there's a lot of stuff, but like, what's on your mind that's like the biggie of like—

I think that the most disruptive thing is where AI is going. We're not done yet. We don't know what that looks like. It requires intentionality because just because you might go to bed with some assumptions about where the world is going, you might wake up in the morning and that's not true anymore. And so I would definitely, I, in my career, I have not seen disruption like this. I think the last one that was even close was the internet. But the funny thing about the internet was, strangely enough, it didn't really boost worker productivity. I think it's very clear this one's gonna boost worker productivity. And as a result, it's gonna be a lot of disruption among our workers. What skills do you need?

Yeah. Change in role, like how you actually build the organization together, all of those things, right? There's a reflection in this besides just the what's typing. It, it's gonna cause change and ripple through there.

I mean, here's the surprise we found. We always hire grads right out of college, but my leadership team has been leaning more in because they're finding, unlike all the rest of our employee base, they're coming out with 2, 3 courses in ML. They are ML ready in a way everyone else has been self-taught in that domain, or they've been data scientists or something. All of a sudden we're able to pick up folks who normally would be very junior who are able to be incredibly productive compared to the more senior colleagues right now. Yeah. It just tells you like how the industry is changing and how things are moving around.

Someone riffed with me on that part of it around, you know, there's been almost a ritual that actually happened in engineering teams around like your progression and training and the kind of an apprenticeship model around as you grow through your career. But with the AI assist and like this whole world, you're describing people coming up ready to rumble, but also the tools can actually help them accelerate too. You know, that probably is going to have some kind of effect around how we kind of evolve our skill sets over time. Like, what does career evolution look like? What does being senior mean if we don't have the training grounds anymore? Because there's the actual coding, but there's all this X-factor other stuff that we do, whether it's debating in the hallway or like convincing people like Those things aren't being assisted yet. Maybe they will, but that's going to have X-factor changes too.

I do think some of the key things that our senior engineers do, like nothing has shown me good architecture design from generative AI, not on a tricky problem. Yes, it can take a general pattern that others know and tell you what it is. That's what's great about it. But not tricky system design. Wait, where's the performance bottleneck really going to be? Can I find it? I don't think this wave of technology is gonna get us there, but it can help you with a lot of the road. But then the other thing also is like the field is moving so quickly that the stuff, forget about us, me, I'm old, but you know, like someone who graduated 5 years ago got a different curriculum than someone who's graduating today. Right? And I think that's having a little bit of an impact.

Well, and to my point earlier, like you're like, where are the systems bottlenecks or such? That historically was learned over 20 years suddenly after solving some issues and bugs. And as you kind of went through your career, does that still exist, that pathway?

I don't know.

Like, anyway, I don't have the answer clearly.

Yeah, it'll be, it'll be interesting. Yeah.

Well, okay. Which takes me, I think, a little bit to the, like, this part of it. Like, actually, one thing I learned, like, talking to Dan, I started this conversation in the beginning being like, what's your job like? Like, what do you do? Right? Um, and, and what I, what I've learned here is that Despite, like, on paper it says your technology, everything you talked about is in this juxtaposition between the business outcome and technology and your customers. You're kind of dealing with a triangle of things there where it's deeply incorporated. And I think, you know, I naively thought when I started my career, you know, Chief Technology Officer, like, that's just you get to make all the tech decisions, right? Like, you do architecture all day. I'm not hearing that from you. It's part of it, but that's not all of it. Um, so if you could talk to, to young Dan Sturman or young Andrew, like, a while ago, like, what do you tell them? Like, you know, what do you tell the next folks to think about? Like, if you could have, you know, given an assist, if you could have been an AI tip to those folks, like, what would you tell them?

Right. Um, you know, that's, that's, that's a, a good question. I think the biggest thing I would tell myself is don't sweat your long-term career plan. Like, every long-term career goal I've set for myself has turned out to be like silly and off. I started my career absolutely sure I would never go into management. Like, that was what I was going to avoid at all costs.

All they talk about is business stuff. Gee, those people are boring. Yeah, that's right.

And I want to stay in the technology zone. There are ways to stay in the technology and do technical things. But, you know, I honestly, I found I was better at management than there were folks around me who were just going to do pure tech better than me. So I just went to where I could help the most. Right? And I think that's worked well. Like, just one thing is keep learning. Like, one thing when I sit back and look at my career, I've changed jobs in a bunch of ways that have been almost always lateral moves, never looking for that promotion, and yet so I could work on something very different. And I realized it was being driven by my sense of desire to learn, curiosity, and so on. And that, I think if you do that and you dig in with relish and you're motivated, then other stuff like promotions and career advancement just kind of come because you just, you're naturally just cranking away and people notice that like, oh, this person's doing a good job on this, let's give them more is kind of what happens, right? So focus on learning, focus on— don't try to plan out too far, but just focus on doing a great job at whatever you're doing. And, you know, for me at least, it's worked out. Like I said, where I thought I'd start my career and where I am now absolutely had no plan to this. Didn't really work towards this sense of having it. I see people have like these roadmaps laid out and I kind of go, ooh, yeah, like, odds are that plan's not going to survive contact with the enemy, right? Like Waterfall career.

I don't know.

Yeah, that's right. You got to be agile. That's a good way to put it. Agile career development is the way to do it. And it's worked out really great for me. I mean, I've had a fantastic career and I've got to do things I never expected to get to do. I never thought I'd be in a company that built a game platform, but when I was looking for my next role before this, Why did I come here? Because I got to work on graphics and AI and systems and economic systems all in the same job. I was like, where else am I going to? Every other company has like one key thing they're doing. Here there was like an open pro— like 8 open problems I could go work on, right? And just that was the most attractive thing to me versus anything else I could go do.

That's amazing. Well, Dan, thank you for coaching young Dan and young Andrew. On this. Like, I think we, we'd all be blessed to kind of, uh, see the things you've seen and get to where you've gotten to. Um, but tying this off, like, I really appreciate being here. This has been super fun for me, hopefully for the audience too. Um, it is a wonderful reconnection for me and you here too, and, and we should do this over a bite of sushi in San Mateo soon, right?

Okay, sounds very good.

Dan, thank you so much. And, uh, with that, I think we're going, uh, back to Glenn and Jasmine. Thank you, folks, really appreciate it.

Thank you.