Friends of data: Daniel Sternberg, Head of Data at Notion

Thank you everyone for being here. And thanks especially to Daniel Sternberg for being here. Um, I am really excited to be doing this. This is the first of a series of these we're gonna be doing. We've done different events over the time, but, um, this is the first of a bunch we're doing, bringing in data leaders and people that we respect in and around, um, the world of data, folks we get to work with, folks we don't. And, um, I am very fortunate and grateful that we do get to work with Daniel Sternberg and the amazing team at Notion as, uh, We are Notion power users and, uh, uh, we're very proud to have you guys as customers and a lot of Hex usage over at Notion. So, um, uh, it was very natural to ask, uh, you to be our, our first guest for the series. And, um, part of that's cuz we get to see up close sort of how you guys use data at Notion and, and, uh, I think the way it, it factors into the way you build the product. And, um, I'm really excited to talk about that with you. So thanks for being here.

Yeah, thanks for having me. Really excited.

Daniel, I thought we could start, maybe just tell us a little bit about your background, how you got into data, your sort of path through various roles, and what brought you to Notion.

Yeah, for sure. So my path to data I think is a very, was a very common path circa when I did it, which was like 2011, 2012, which is when I finished my PhD. So I am the—

Wait, what was your PhD in? You gotta tell us that.

Okay, my PhD was in—

You can't just drop that in the middle.

Yeah, my PhD was in cognitive psychology. Like some other data scientists I've worked with over the years. And I studied learning, I studied like neural networks in the worst period of time to study neural networks in the like mid-aughts. And neural network models of human cognition. So I did a lot of really boring video game type experiments with people and then I built models to model the data from those experiments. You can naturally see how that probably took me into like a data science type role over time. And I, you know, at the time I was finishing my PhD, I kind of like looked at it, this is my life advice moment at the very beginning. I looked at, when I was trying to decide what to do next, I didn't really have like career goals 'cause I was already deciding I was leaving academia and I didn't wanna stay and do that after my PhD. But I, so I just, I didn't really have real career goals. I was just like, well, one, I wanna have flexibility where I live and do what I want in that sense. But otherwise, what am I doing right now that I like doing? And what am I doing right now that I wanna do less of? And can I find myself a job where I do more of the parts that I like and do less of the parts that I don't like? And I actually think it's good to have long-term career goals, but also like most of your life is just doing the stuff that you have to do every day. So like, if you don't like that, that probably sucks. There'll always be things you don't like to do. So for me, it was like, I liked doing experiments to some degree. I liked the analysis and modeling part a lot. I didn't like writing papers. I wanted to get tighter feedback loops on what I was doing. And I kind of just got lucky. I was out here in the Bay Area, I was at Stanford, and I ended up going into data science. So I kind of made my way down the stack over time. I got a job at a startup called Lumosity, which is a brain training game company. You probably got inundated with ads from them probably like 10 years ago. And it was my first job out of grad school. They, I had a leg up there. I actually like didn't get accepted to a variety of like places you would want to do data science in the year 2012, like Facebook or wherever that was like the hot place to go at the time. They did not accept me for a role. I got a job because they were interested in my domain background and I was good enough at stats and analytics stuff. And so that got me in. Winning combo.

Winning combo.

Yeah, exactly. My experience to them was actually relevant. So that worked out. I was there for 4 years. I ended up managing a small team there. I got to work on a ton of different stuff. I made lifelong friends. It's kind of like the first job out of college for some people, although I was 28. So you get, you make friends. And then I was there for 4 years, probably should have left a little sooner than I did. Went to Gusto after that. I was their first data scientist. I was still an IC doing data science work, but I was managing, 'cause I had some management experience, like a year and a half of it.

That's enough.

A tiny data team.

So then you run a team, yeah, go for it.

Tiny data eng team there, which I had no business managing. Don't tell anyone that. And ended up spinning out data science and building that team out for a number of years once we started, got some momentum there. Um, and then my last year or so there, I ended up managing the whole, uh, data org. We kind of did a big reorg. We centralized teams that were previously actually more decentralized, and I was leading those teams for my last year and a half after having built really strong relationships with people. And then it was just time for me to do something new, and I came to Notion, and I can unpack more why at some other point in this conversation.

Cool. Well, that's a good segue because I do want to talk a little bit about Notion.

Yeah.

Tell us, like, how does Notion kind of work? As a company and how does data fit in?

Yeah, for sure. So Notion was the first place I've ever been, I'd worked at myself where like the team started in a very centralized way. And I think the reason for that was one, Notion, there's a culture around like really respecting craft. And so it leads the company toward having more centralized structures around these things. It's like, okay, we want data experts to have a team that works together. I'm not saying it scales like that forever necessarily, but that's sort of the setup. Like our design team is set up that way, various other teams are set up that way as well. So we started that way and today, what does it look like? Where does data fit in? We are purely like functionally organized company. So I've been overseeing for the last 2 years the overall data org, which includes data platform, data science, data engineering, BI engineering, which is part of our data eng team, and now, also AI as well. And we report into the CTO. So we're part of the technical org, but parts of the team support some of the business side teams as well. So like support our sales team, support marketing team, support finance on the data and side in particular, and increasingly on data science as well. But Notion is operating this way, yeah, pretty much from the beginning.

And so there aren't people with data titles like within finance, within marketing?

There's like a couple. Where we've done that.

Rogue. Rogue.

I actually like, people go nuts about like centralized versus decentralized and like I've been in versions of both. And I think a lot of those decisions are, I was talking to someone about this right before this actually. Some of those decisions are related to like, what is the right structure given the, honestly like the business model and the type of product that you are. Um, and for Notion, I think one of the reasons we tend to be more— other reasons we tend to be more centralized is that Notion, like, at its very core is all about being like one holistic product that can do many things. And so like being together is actually really important. Um, and so like, I think you guys have like one SKU. We have one SKU. I mean, we have like, there's, there's different tiers, but you're buying, when you're buying a Notion, you're buying Notion. Exactly. Um, and so we're orient— we've been set up that way pretty much from the beginning. So yeah, it's, we have a couple of places where we've had like individuals and I'm actually not opposed to that structure in the right places. It's just so happened though that every time, almost every time we've looked at, hey, we're not really supporting this area of the business or this function or something like that right now. And we talked to the partner team we're working with on like, okay, we're gonna get one person to work on this problem. This came up with like, I'll use like growth marketing as a result. Like we didn't have dedicated growth marketing support. And the decision usually they're like, no, I don't actually wanna like manage that person. Like you all should manage them.

You take 'em, it's fine.

Yeah, it's fine. As long as they're like supporting my teams, like I don't care. And I think like that to me is a signal that like there's decently high trust in this team and that like we hold to our promises of like actually supporting them and not going off and working on random pet projects that the data team has. So I usually feel good about that.

Well, this is a common failure mode of like the ivory tower. Our data team and, and, um, I will come back to that topic in a moment, but I want to ask very specifically, Notion, I think the thing that Notion maybe is known for best, um, is that sense of craft you referred to. And I think there's a sense of taste, a sense of opinionation. And I know from, uh, a lot of exposure to people at Notion and just, I think you can just tell working on the product, there's, there's, there's times where I sense that you guys are incredibly data-driven. And the data is driving the decision, and there's times where it's like, I don't care what the data says, we're doing this 'cause it feels right for the product. And I'm just curious like what that means for being a culture that has that level of sort of taste and intuition and opinionation around the craft of the product. What does that mean? Like how does that affect how you think of how data fits in and how you're partnering with different functions?

So it's actually funny. It is like very core to Notion. You're exactly right that like craft is really important and like taste is really important in the how things get built. It's funny, my last company too, I felt this too. I think actually like any, any like the business we're in, I think it relates actually to the business model in certain ways as well. So business we're in, like at its core, like, you know, hopefully you've all used Notion and you know like what the value of the product is. But at the end of the day, like, okay, what is the business model? Well, you're like a SaaS subscription business essentially. And like, you actually can't rely on data alone to make feature product feature decisions in a SaaS subscription business because you actually can't really build the, and I know this 'cause I have the scar tissue and tears of trying to, you cannot go from, oh, if I build this feature, it will drive this usage, which will drive this revenue in this like very clean, clear way. Like you maybe could at like, I don't know, a big social media company where it's like eyes are money. Like, eyes aren't money at Notion. Like, value is money at Notion. And how do I prove value? And that's like a complicated thing to prove. And so like, I think where we use data, you know, we don't use data to tell us, oh, we should refactor the sidebar in Notion in exactly this way. Though maybe we should refactor the sidebar.

I have thoughts on that, yeah.

We use it to help us like have some intuitions about what are some of the problems. And then we obviously get customer feedback from our research team, qualitative feedback as well about what are some of the problems. We have like designers like go and design like different versions of how they could change this. And we play around with how they feel and think about it. We dogfooded internally. For a while. And then we do have data scientists think about how am I gonna measure, sometimes hopefully I can say measure if this was successful or not. And in some cases with that sidebar example, actually more of the time you're like, can I just measure that I didn't screw things up too bad and get some decent customer feedback? Because it's hard. Like it's hard to be like, oh my gosh, I can show you that I've like driven engagement in Notion up by X amount because I changed the sidebar. Like that's really hard to actually get consistently. And so in my experience, like what you find is like a useful framework that I've used on like data teams for myself in my last two roles at least has been, is the team I'm working with a team whose job is optimization or is their job to like build new stuff that solves problems for customers? Their job is optimization, it's going to be very data-driven. You know, growth teams tend to look more like this. Another example at Notion that has some examples of some of this is like our search team. Like, search has pretty measurable things they can look at to be like, did I improve the experience of search? But like, our docs team, like, building like how we improve the, like, docs experience in Notion— don't get me wrong, they can have some high-level goals that they're driving toward, and they can make sure to not try to hurt stuff, but like, they can't measure like every feature they launch. Did it do— did it like drive the product forward in this way? Like, there's just no way.

Yeah, it Like it's, you guys just launched suggested edits, which is like the big feature I've been like dying for forever. It strikes me it'd be very hard to have like the data suggest that you should build suggested edits. You can measure like, does that change engagement? Does it change collaboration patterns?

But yeah, it depends on what you count as sales data. I mean, sorry, I skipped what I was gonna say. Depends what you count as data too. Obviously there's other things you do if you have an enterprise side of your business, like Notion does, you know, you look at like what are some of the big sales asks coming in, those sorts of things, which are quantifiable.

Yeah.

And you're like, hey, like, A lot of people need this. And then the taste part is, okay, how do I solve that problem?

Right, I mean, we have this, we build data software, obviously. Yeah. Like we have this tension internally. It's like you try to quantify things and sometimes I'll look at it like, yep, that feels very clear, that tells the story. And sometimes like, eh. Yeah. We're not gonna do that, we'll do something else. And you have to have both. It's a good segue to, I think, a very hot topic with data teams and data leaders and basically every, single data leader I've ever spoken to has some version of this conversation, which is data team ROI. This is like the— I actually, my theory is this is actually the thing underlying almost every other excitement and anxiety around data teams, which is like, at the end of the day, you're a service org. How do you measure the ROI of that service org? And it's— I wanted to ask how you think about that, because we just had this conversation about how certain things could be quantified, certain things can't, how you guys are organized, and how you even mentioned, you know, someone not caring necessarily whether the data head counts like in their org, but as long as they're supporting them.

Yep.

How do you measure the ROI of that? Do you? How do you think about that?

Yeah, I mean, to be very clear, I do not have like a dashboard where I can see, you know, these are our success metrics for the data org. I mean, we have outcomes or, you know, OKR-type things we want to drive in a given, you know, quarter or half or whatever. But I don't have like the, You know, I can prove to you that data drove $4 billion, $16 million in revenue last month or something like that.

But you know, anyone who says that they do have that, I don't believe them.

No. So I think there's like cases that are like very virtuous and like exciting that come up where there is some insight that you had that like drives a very clear outcome that you can point to. Who I won't name in this audience, they worked with a then Biz Ops person, now PM on, realizing something about how we had set up pricing and packaging in Notion that like it was clearly making it more difficult to, um, there's like a combination of an intuition behind it and some data, um, which had to do with, um, we want like a big value prop of Notion is collaboration, um, like using Notion for collaboration. And we had set up a paywall in Notion, like the way we did paywalling in Notion such that for a certain segment of users, when they created a new Notion workspace, the moment they added a second person, they got pitwalled. Um, and we changed— we, the, the, and like, we saw that like in the segment, like it had a big impact on, not surprisingly, how many of them like added more people. And then as, and then we also had like lower than expected monetization rates from those people. And the insight was, well, what we should do instead is we should actually set a limit around like how much content they put into Notion after they add people. And once they hit a certain amount of content in Notion, then we'll do this. But at that point, they will have actually gotten value out of the collaboration. And like, that's a great insight, but that doesn't happen all the time. And so I think like, I think like it can be exciting. You know, one of the things you can want to do is like, oh my gosh, I can show that with data, we found this like conclusion we would never have found otherwise. Without like some amazing data person.

The magic nugget.

The magic nugget of information. And I think one of the things I take from my background in psychology, and a lot of this happens kind of after I left, shortly after I left, is that, I don't know if any of you know this, but if you followed like research psychology, like experimental psychology, et cetera, over the last 10, 15 years, you probably know that there was like a gigantic replication crisis. And like, A lot of that, some of that was driven by like, it was driven by practices in the field, but why did those practices in the field exist? The practices in the field existed because there was a lot of incentives in the publication process to come up with like really cool, interesting, counterintuitive findings. And the fact of the matter is most things just aren't that counterintuitive. And in a bit—

It was like the whole power poses thing, right?

Yeah, power poses.

You gotta power pose right now.

Yeah, power poses was a big one. It was like, you know, there were some ones I really liked in my side of the world where it was like, I don't know, like, if you like carried a really heavy backpack, then like, anyway, there's like all kinds of really weird findings, or like the height of the room affected like the number of creative ideas you came up with. None of that stuff replicates. And so anyway, I just thought like, I just find that in general, like, there's a lot of things you do on data. It's worth differentiating between these types of things. There's like exploratory findings that really shape how people think, Um, and like show, like showing over time that like we know how to measure our product and like I have ideas about behaviors I wanna drive in the product that will help us, um, drive more usage of Notion and ideally more revenue over time. And like what are some decent ones of those that like people get confidence in, they can drive, et cetera. That's awesome. Then in other parts of the business, that, that takes a lot of work. It actually is really hard and it takes a lot of time, but like a lot of that is like, you see the ROIs and you see the, not just data, but like the product teams and others working more effectively and working with higher confidence in the decisions they're making because of data. So some of that is gonna be like qualitative and it's gonna be like, you know when you see it. Then the other side, like there's a lot of stuff in the data world too that is like nuts and bolts, just like has to get done. And it's important and you didn't have it before, you know, like we need to be able to count how much money we're making accurately.

Dude, counting things, underrated.

And like counting how much money you're doing accurately is like, it's not like you don't reinvent the wheel on how to do that.

Yeah. But doing it well is really important.

Doing it well is really important.

And if you don't pay attention to it, you're going to do it poorly. And then that's bad.

As I said, like Notion, like, you know, the enterprise side of our business, the upmarket side is like, you know, it's a, It's like a sales-driven, that part is sales-driven subscription business. Like there's core stuff you build for sales teams. Like they just need their stuff. And like, yeah, there's probably something that's special to the company 'cause you need to give them certain insights from your product data or whatever. But like, we should not think of, you'll get better ROI if you don't think of the job as a glamorous job, is what I've learned over time. And that's okay.

You just burst everyone's like dreams. They got into data 'cause it was glamorous and you just— Sexiest job of the 2010s.

That was when I started in data science, that was the year that it was like sexiest job of the century. And I was like, of the century? What was the sexiest job in the year 1912? Was it like—

It's actually, what would it have been? I don't know.

A joke for my friend at the time was dirigible pilot was like a good example.

But anyway. Model T salesman.

Yeah, exactly. So, you know, Don't try to guess what the century is gonna be like in, you know, 12 years in the century.

Yeah, but I mean, I think there was, I actually think the, it was funny 'cause the like, sort of like counterintuitive, but actually like non-replicable sort of study findings and this fetishism around data science in the mid-2010s, I think were part of the same thing, which I think in some ways had its heart in the right place, which is like, hey, if you can actually tap into all this data and learn more about the world, like you can probably do, We can learn things and make better decisions as part of that. I think that the synthesis to find is where is that actually going to be highly leveraged? And to the question I asked, where is that measurable? And you mentioned something just a few minutes ago of some of the work is just making sure the product teams can just measure things and see things and have a better understanding. You're never going to measure the ROI of that quantitatively. But unfortunately, it's like a snipe because I think it's almost this preferential— Like data teams spend all their day quantifying things. So when they get asked about like data team ROI, they're like, oh, maybe I can go quantify that. And it's like this trap. And it's like, they're gonna go try to build the best model to quantify that. And it's like, it's just, it's never gonna— You're never gonna put an ROI model in front of a CFO and they're gonna be like, yep, checks out. It's always gonna be skeptical.

If there's a data leader here who knows how to do this really well, I would love to learn from them because there—

I ask every data leader I talk to and I've never heard a good answer, I'll tell you.

No, I find we have like, possibly, not necessarily, there's other teams that are in a similar boat, but we have one of the like, whenever I do a like planning cycle and I'm like writing out OKRs for a team, and this is true here at Notion, it was true at Gusto too, like I'm always like embarrassed that I have like the least quantifiable success metrics.

You're supposed to be the numbers guy.

Yeah, exactly. The least quantifiable success metrics. I mean like there's maybe like, I don't know, 'cause like infra is way easier and like, not easier, sorry, infra's really hard to do. But like, you know, you're like, oh, I care about uptime. I know what that goal is. I'm trying to improve reliability by this amount. It's hard. I can look at things like we have an initiative right now trying to get more self-service with data, which probably you wanna talk about. And— Let's talk about that. And there I'm like, okay, what can I do? I can probably get something like we have this many weekly active users of various data tools and we want to expand that number. And I have some view of like, some qualitative metric that I'm getting around their confidence and ability to use the tools. And like, that's kind of it.

Like, how am I supposed to do, like, uh, my, my, my version of this is it's about, uh, NPS or net promoter score, which is like a way for companies, for those who aren't familiar, to measure like, would you, would you recommend this to a friend? And I always think that the real way to measure data team ROI is to like go to your functional partners, go to marketing, go to sales, go to product and be like, would you give up a headcount to hire another data person? Or what would you do? What would you give up if we took your data person away? You're actually really trying to push on like, is this actually like a tangible, measurable value? And I think it clarifies the prime directive of the data team, which is you're trying to like move the needle for that partner.

One of the things I really liked in my last year in my last job was we got, I didn't necessarily like the reason we got there, 'cause there was some org structure changes that like were tough, but like, we got to a place where we actually let like a product team, this is on the product side at least, like they could just go, I have this much headcount, do I want, like, do I want a designer? Do I want an engineer? Do I want a data scientist? Do I want a PM? And obviously like they'd have to go to me if they wanted a data scientist, I have to be like, oh, I have a manager who can actually manage them and like figure out how it fits into the org or whatever. But like, they actually could make those calls. I was like, I don't wanna make those calls.

Like actually at the end of the day, you don't wanna be the one campaigning for headcount.

They should be asking for it. 100%. 100%.

They just provide it.

Yeah.

I, I totally agree with that. Yeah. I wanna, I, I got, um, one more question and then a quick question and then we'll open up to the floor. Yes. You mentioned, um, earlier when you were talking about what rolls up to you, you mentioned AI is like a, yeah. A last thing you're like, oh yeah.

And AI, it's actually a really great team. I don't know if any, that's probably the one team not represented here by Notinos here tonight, but that's fine.

But, I'm— what?

Busy working. Yeah, they're busy working.

They're busy working.

That's right.

That's right. Um, shout out to the Notion AI team. Um, but I want to hear, I want to talk a little bit specifically because you guys have built a bunch of AI features into Notion, but also like how AI will change or not change data work is obviously also like a big topic of conversation. It's a topic of conversation everywhere right now. And I think within data as well. How are you thinking about that? How are you looking at that? What have you come to believe or, or what are you skeptical of?

Yeah, I'll start with like the tangible, like what have we actually done? So putting Notion AI features aside, I hope you've all tried Notion AI, our Q&A is amazing. But I think the only place where we've like started using AI meaningfully on, for like data from a like using data to make decisions standpoint is we started using an LLM to help with first pass of documenting all of our data models. And then we review it. We have human-in-the-loop review of it, but it's actually been really helpful to just like, that's one of these like bootstrapping problem, like cold start kind of problems where you're like, oh my God, like nothing is documented. And like a company can literally get to like having like hundreds to thousands of data models before they ever do any documentation.

Very, very familiar with this, yeah.

Yeah, probably you deal with this because your customers don't document anything and then they don't have access to any, and then that makes TextMagic not work as well, et cetera, et cetera. So we bootstrapped it.

Welcome to my life.

This way. And it's actually been really useful. And it's one of the reasons I actually feel like we can invest more in like self-service on our teams now is because actually there's a lot much like documentation is an order of magnitude better than it was like a year ago. That's like it. Like we don't do a lot with AI in data work otherwise right now in my experience. Feel free folks to correct me when you're talking to people afterwards or raise your hands. But, What do I believe? Well, using that as just jumping off from that example, I think the concept that like, the concept that AI can help accelerate the work of people with some basic data skills, I think is uncontroversial in the same way that like it helps write code, helps you as a copilot when you're writing code. It helps for some people when they're writing too. So I think that's like uncontroversial. I think the, we're gonna get to a place where, where like I can just ask a question and it's gonna look up everything for me and answer the question for me for anything mildly complicated. It's not that it's not theoretically possible, it's that it mis— it like, it misattributes where the difficulty is in the work, in the data work, because the path you, like I mentioned this documentation part, We had to have people who have thought about what are the right data models to build. They had to build those data models. In some areas, that's easy, that's like more cut and dry. Like you can probably think about what the right data models to build for revenue stuff is to a certain extent, if you know how to do it and you've been versed in that area. But on product, it's like different at every single company. So like what are the right basic data models? Then you need to like actually go through the process of like, getting to a decent state of governance and documentation, which requires humans to do. And like, that's most of the work. And then you're like, "Oh, I want to ask this, this like simple question." It also needs to be like a really well-formed question, which like a lot of, on the data science side, you spend a lot of your time, I think, like helping people ask better questions. Maybe AI can help you ask better questions over time. But so much of the, like, to me it's like the company that is able to like automate out a lot of that work, is a company that already had like a sizable data organization that like helped bootstrap their way to like building all of that underlying stuff. So until you can get AI to build all of that for you, you're probably not gonna be able to do it.

Yeah, I completely agree. And I think where we, there's like some motivated reasoning in that we sell a data software tool that has AI, but it's like also like it's meant to augment humans. And so I think, there's a degree to which this is like talking my book. But at the same time, I think I don't see how— I think if you primarily understand the role of a data scientist or a data analyst is like writing code, then yeah, like LLMs can write code now. And like, then it's like, sure, but like, what code are you writing? Why are you writing it? How does it fit in? How are you explaining the results of it to a stakeholder? How are you helping them better understand the questions to ask? How are you hearkening back to the last conversation you had about something? There's, there's all of these things that actual like really great data practitioners do that I don't think any LLM will ever be able to fully supplant. Just as within Notion, it's not like Notion's AI writing tools are just going to magically write all your blog posts for you. You still need to be like thinking critically on like, what should we be writing about? What does our audience care about? It's not going to magically like just produce all of that for you. The human is still like the heart of it in my estimation. But— Cool. Cool. Last question. Do you have a book, article, anything that you really recommend to people who are in and around data that has informed your thinking or worldview?

Wow, okay, that's great. I will be honest, I don't read a lot of like just data books these days.

I probably have some. Doesn't have to be just about data.

Yeah, no, I was gonna say, that's what I was gonna say. I mean, there's like various Substacks and things like that that I enjoy that I'd read in the data space. I enjoy Ben Stancil's one 'cause he kind of goes all over the place and he tries to be snarky. But I'm gonna give you like a very out of left field thing, and it's more of like this has influenced me, and like if you don't like this sort of thing, don't do it. When I was in grad school, I got really into Ludwig Wittgenstein, and like his, 'cause I worked with a professor who was a little too obsessed with him. But anyway, not my advisor, different professor. And like, One of the things that I have tried, like I am affected by as a leader, I think is like deeply thinking of problems as being very contextual. And like most situations I deal with with people are not issues of like intrinsic aspects of the humans I work with, but like the situations in which they all find themselves. Anyway, Philosophical Investigations by Ludwig Wittgenstein, like has influenced my life and thinking in a lot of like weird deep ways. It is not a thing that you should pick up if you're just like casually hanging out with your friends. Though it is, you know, you can read it in bits and pieces 'cause it's like short sections.

And what's the core takeaway? Like what's the sort of, what's the maybe like moment or a time where you find yourself reflecting on that?

I think it's when I, it's how does it come up in my work life? It's so deeply embedded in me as a human that I don't know. I think it comes up for me when I am trying to talk to people with very different backgrounds, and like, I have to do this as a leader a lot, and like different functions and areas about similar problems, but like completely restructure the way I talk about it to depend on that specific person. So I think it's influenced me a lot of ways in like catering how I interact with with different people with this concept of like language is all just about use and like the meaning of words is not an intrinsic thing. And so it's kind of like figuring out how to do that.

Yeah. I'll ask ChatGPT to summarize this for me later so I don't have to read the whole thing. Yeah, that's right. Yeah, as a SaaS platform, collecting data can be hard. I think it's true sort of universally. And the question was like, is there ever a time where you have to generate data to fill a gap?

That's a great question. Honestly, I can't think of an example of that.

Huh?

Where it's come up for us. Has it come up in Hex?

Where we generate data?

Yeah.

We generate a lot of like demo data. That makes sense. Yeah, I mean, this is more specific to Hex.

Yeah, I mean, we have to generate like fake content for Notion for certain things, like certain features that we're testing and things like that. But one of the things that's an advantage for us, I think what makes Notion a little different, our business looks like a SaaS business. Notion is, we are a product that ideally it's wall-to-wall across your business or you're using it personally. And one of the nice things I mentioned earlier is that we can dogfood our product like crazy. So we don't have a lot of situations where we need to do that. It's come up for things like we're building AI connectors with other tools now. And so like, uh, if we're, um, check that out, we have Slack is out in the market right now. So you can connect Slack into Notion. You can do AI, like Notion Q&A on Slack, uh, on your Slack data. I don't know, you, Paul, probably have access to that.

Oh, we've had access to it for a while.

Yeah, that's right. You were one of our early ones.

Alpha.

Yeah, you were in alpha. Um, but there are some tools that we're testing now that like Notion actually doesn't use very much. And like in those cases, we're like, oh, I gotta generate a bunch of content and things like that. So it comes up in those cases.

Yeah, we've been, I guess, We've been seeing some of this with, similarly with product development around AI is like there's certain things where we need to generate like a lot of evals. And there's like a set, even for stuff, even though we dogfood our own product, it's like we gotta then like turn that into evals and canonize that. And then we also have to think about where are we actually representative of our customers and where aren't we? And that can be, it's always a risk of like you build for yourself and then you like get it out in front of a customer like, oh, you operate completely differently.

One of the things we've leaned into there, which, you mentioned already being part of is like we've leaned into some cases, like we get some early design partners on some of these features so that we can like at least get a, like, once we feel like, okay, it's working for us internally, let's get like 8 customers using it. Um, who are like comfortable with a maybe crappy experience. Um, and it makes a huge difference.

Cool. Other questions? Jess. Jess asked about the history of the data team at Notion, how it's evolved. Big inflection points.

Great, great question. Um, so I joined Notion about a little less than 2 years into its data journey. So I was not there from the very beginning. And first person who joined Notion on data in early, in like spring 2020, was a data scientist who actually managed a data science team at a prior, at Asana before that, who many of us know quite well at Notion. And She— we hired— this is the only company I've been at like this, even though I know this is a common trope in startups, which is like they hired a few data scientists first because the data scientists are the jacks of all trades at the time and they can get their hands dirty with various stuff. And the tool stack had gotten a little bit easier to work with, at least. And then there had been a head of growth at Notion who was not even on— this person's no longer there either— who was not even technically a data person who actually stood up a lot of the initial stack. 'Cause it was like, oh, I'll just stitch together Fivetran and set up, turn on Snowflake and like get it out, get going out of the gates. So like our first few hires were data science folks. Then we started building data platform. It was like, oh, we need someone to actually own the infrastructure part. And actually the data eng side came last in terms of those, well, other than AI, in terms of those teams. And so it kind of started from a, let's try to get some value out of data really quickly. And then, by the time I joined, we had a data eng team of like 4 people for like 4 or 5 months. We had, how many data scientists? I don't know, probably like 8, 9, 10, something like that. I don't know, I'm looking at people who were here before me. Lisa, who's back there, is one of the first few from that first cohort before we even had InforPeople. And then, And then, like, one of the things that came up for me evolution-wise was just like, I felt like within 6 months of my joining, I realized that, like, we had grown the data science side a lot, data platform and the infrastructure side is going really well, but we started to feel really bottlenecked by not having people in a good position to own the core datasets that were being heavily used by lots and lots of people. I think a common failure mode that you find, it's like, it actually is very reasonable to me that you, like, Hired a data scientist first, I would probably hire a data scientist and some kind of savvy data engineer/data platform person. Get those two people together, that's probably the best way to go out of the gates. But people don't always know they need that. I think what you find though is data science tends to be kind of forward deployed. And so it's great out of the gates, they can build some datasets in dbt and with the technology now it's easy enough to get some things going off the ground. But once you have many teams using the same data models and the same datasets, data scientists who are focused on whatever their partner is thinking about are not the right people to own and maintain those things long-term and think about how they should evolve and be developed. And so at that point, we actually kind of, I don't know if we ever truly stopped hiring data scientists. I'm sure there was a non-zero number, but we slowed down there and I actually reinvested a lot of that headcount in building out our data engineering team because it was just clear that that was a bottleneck at that moment in time. And you could see it in literally engagement survey feedback from data scientists. Scientist who was just like, I am responsible for every step of the data lifecycle. Like, this is not sustainable.

Yeah, you're doing, you're having to build the whole basically stack every time you're, you know, of all the data you need and all of that.

And so that made a really big difference at that point in time. And so we built a really, like, we grew that team a lot last year and that's why I think we're in a position to like invest more in self-service and things like that, which would've been really hard before then. And then AI came along later because, We got really excited about some things we were testing out with an early version of GPT-4. And then you can look this story up online. You don't have to hear it from me. And then kind of we started incrementally investing in AI. And now actually our data platform team spends a significant portion of its time thinking about large data processing problems that are upstream of our AI use cases.

Yeah, well, let me ask about that. When you said sort of AI is within data, Break that down for me a little bit, because I assume there are people who don't report up through you who are like product engineers working on AI features. How do you guys partner with them? What is that sort of AI stack? Does the data team own what is living within sort of engineering per se?

Yeah, so the way it works right now, we started actually the AI team in a very centralized way as well, but that was for probably a slightly different reason, which was At the moment that we were building, like actually like said, like we're gonna move to have an AI team, the first thing we did actually was just like borrow a few people, engineers, to work with one of our co-founders for like a few months.

Same here.

So that's fun. And I donate, you know, I helped like donate like the time of one engineer from our data platform team for what eventually became Q&A. I take no credit for Q&A's existence. I just, you know, helped like make sure we had the people.

Daniel, the father of Q&A. You heard it here first.

But once we decided we actually needed a team, we actually wanted to enable that team to be pretty disruptive for a little bit. And so what we did was we said, we actually want the full stack on one team. So like we moved a couple people from data platform because of some of the things we were building with embedding, looking around with embeddings. We like moved a couple product engineers. We have an amazing leader for that team who came over at that point 'cause we actually needed a manager. And then we also, like, we had some UX people, and then we had just hired, like, a couple ML engineers as well. And so actually, they were all together. And so, for the most part, up until a few months ago, actually all the core product engineers working on, like, the core AI features were on this team. What's happening over time is, like, you know, the balance you have to strike at a company is, like, you wanna move fast, and, like, moving fast was one of our advantages, and moving fast, being centralized like that was better for moving fast, 'cause you weren't worrying about, Oh, I need to get like 4 different PMs to be thinking about this thing, et cetera. You know, we can have like 1 PM for a while thinking about this now, maybe a couple. But the problem over time is like, you can create a really wonky experience for users if you like, aren't like thinking about all of the product surfaces holistically and all of that. And so what we started to do is try to like actually start hiring some people with AI backgrounds who are part of those other like basic product, like key product teams. But actually many of them like start incubating on the AI team and then maybe move over later. And then we're gonna seed this over time to like make a transition so that more of the just basic AI product engineering is happening on other teams. And we're kind of like, I would say a third of the way on the journey to like decentralize it.

You guys within data will continue to own the sort of like some of the infra pieces of that and then they'll build on top of that.

The infra pieces and honestly probably for the time being the ML, like the more ML-centric pieces, 'cause I think that, That has other implications on talent strategy and like, are you in a good position to have product managers managing ML engineers?

Well, it strikes me as similar going all the way back to very early in our conversation, like the question about like centralized, decentralized data teams of like, do you want the finance team managing a bunch of data people or are you gonna manage them sort of on the finance team's behalf? Yeah. Although they're primarily working with finance.

Yeah, it's exactly, I think with ML, it's like for that type of role in particular, Until you get to a certain level of scale, I think it's very hard to decentralize that team.

I agree. I joke all the time. We have a very similar setup. And I say, I sometimes think of the role of our product, of our AI team, especially the product engineering side, is to disband the team. It's like, over time, we should get to the point where the AI features are easy enough to develop on. We have the infra. And it's like, there's a lot of work that needs to be done just in this space generally to make that possible. Eventually you shouldn't have one team that's building AI features. It's gonna be more ubiquitous. It'd be like having a standalone team that's building cloud features in 2015. It's like, that made sense then. You wouldn't think to have that now.

100%.

Same transitional happen.

Yeah, I think that's exactly right. And I think like, there's like platform components. I don't just mean like the pure infra. I also mean there's like, you know, there's gonna be some like, you can think of it as essentially core APIs for your teams to work with that are probably gonna be managed by a centralized team for some time. Just because there's a lot of specialized skill set that you need in that area. But like, yeah, should you be able to train more engineers who are like on any team to do prompt engineering? Like, obviously.

Cool. We'll take one last question and then we'll, we'll let everyone go. Sai. So Sai asked a really good question about experimentation. How experiment— you know, running an experiment takes time versus just doing something. How do you think about what needs to be run through an experiment and what doesn't? What needs to be tested and what doesn't?

Yeah, that's a great question. Okay, I think there, I actually kind of have a framework I've thought through for this over time for having this conversation enough times with people. I think there's like—

Not the first time you've been encountered this.

Yeah, there's like 3 types of use cases I think of primarily for experiments. I mentioned earlier there's like optimization teams and like building teams and like on optimization teams you do a lot of experiments. And that's because in fact almost everything, you shouldn't say, I will not say everything. There are things our growth team does that are not experiments. There are times they don't. But a high percentage of the things you ship are going to be experiments because you probably have a very explicit metric goal you're trying to drive. And like, you're— there are probably user experience considerations in some cases, but like most of the time you would go with the thing that just wins on the metric purely on its own. And so in those cases, I think you're like generally going to do experiments a very high percentage of the time unless they're— is some like, we must do this for compliance reason, or we must do this for, you know, name some other like critical business reason that's separate from your normal focus area. Second bucket I would say are, I want to, like I have a disagreement on the team between like, should we do it this way or that way? And we can't come to a conclusion.

Which shade of blue?

Which shade of blue? Yeah, hopefully not. Um, and so, but like, you, I've seen this where it's like, oh, this, you like, we could do it with like this type of UI or this type of UI instead. And like, we're not sure, so we'll just try both. Um, I think some company, like that one, I actually like, I choose, I've chosen now, like very much depends on company culture, whether you're comfortable doing that or not. At Notion, we actually don't do that very much.

That's not surprising.

Um, like there's just opinions and that's fine. And like most of the time I'm fine with that. Um, and I don't fight that battle. Um, and then the third bucket to me, which is the other one that comes up, I think, on product teams, like, really, like, basic product teams sometimes, um, and you just need to think about if it's one of these cases, is there's this thing I'm doing that, like, changes fundamentally how people use the product in some way that is, like, gonna be, like, non-intuitive to understand. So what I wanna differentiate between is, like, you mentioned suggested ads. We didn't need to run an ex— I hope we didn't, Pam, tell me if I'm wrong— run an experiment on suggested edits. Maybe, like, I don't think, like, we were like, we tested it internally. We're like, it's good. We'll get some user feedback on it. Cool. We know people, like, we think people want this feature. So we shipped suggested edits. When we do make side— like when we were doing like sidebar changes or we launched, um, this might— if you've seen it, like Home in Notion.

Yeah, it's pretty sweet.

Um, but it's just like a new landing page inside of Notion. That like fundamentally changes theoretically how people navigate the product. And like, I don't know what that's going to do. Like, I hope it will be helpful. I also hope it won't be harmful. And like, I should probably do a holdout or something so I at least have an idea of what I did and can make a decision going forward. And I would say on product teams, like most of the experiments we run are either because I'm trying to learn something and just like I want to get data because data has value in this case. But I'm not trying to optimize and make a call. Or it's because I'm doing this, which is I'm trying to deal with something that has weird, complicated causal impact on how people use the product. But probably on my docs team, I'm not generally going to be that because I don't know, does it feel good? Probably if I think it feels good to use docs in this way, just for inside of a doc, And like most of the users they try it with, they feel good, it's fine.

I think it's a good place to end it. Thank you, Daniel Sternberg. Thanks to everyone for coming out and for the great questions.

Awesome.