An AI Assistant to Work Faster with Notion's Head of Data, Daniel Sternberg

All right, Daniel Sternberg, welcome to Data Driven NYC. You are the head of data at Notion. Notion, of course, is a highly successful startup that serves as an all-in-one workspace for note-taking, task management, and project management, as I'm sure a lot of people here are keenly aware. The company was started in 2016, which actually feels like not that long ago considering the success of Notion. Notion has raised about $340 million in venture capital. And before Notion, you were head of data and data science at great companies like Gusto and Lumosity. So I'm very excited to have you. I've been looking forward to the conversation. What is the— what does that mean, the head of data at Notion?

Yeah, for sure. And it actually means a little bit, what it means is a little bit different than even when I joined about 9 months ago. So Notion, when I joined Notion, we'd had a data team for some period of time already for a couple years. And the data team at Notion looked pretty much like you would expect, I think, at a lot of companies like, like ours. So data science, data platform, data engineering. Um, and, um, and then about, so, so that was, that's the historically pre this current AI focus. Uh, that's really what the team was. That was the team I, I joined back, um, in the last summer. Um, and you know, I think in that, putting AI aside for a second, in, in that space, I think Notion is in a, particularly interesting position that is what got me excited about joining the company in the first place, which is Notion. Hopefully many of you have already used Notion. I hope we are not just a, you know, here's another buzzword, product-led growth, B2B SaaS type business. We also are— we have a very, very large personal user base. This differentiates us, I think, even from a lot of companies that are like the flagship companies of PLG in that we actually have a very, very large personal user base. And so while our business model and business looks a lot like a B2B SaaS business, we have this— this is what got me interested, one of the things that got me intellectually interested. We have this very different, almost consumer-like availability of data about how people use our product. Because if we're doing our jobs right, whether you're a personal user or a user at a company, Um, you are, uh, if we're doing our jobs right, you are using Notion day in, day out, right? Really, it's a core tool, you know, the way we use it at our company, uh, for collaboration, for knowledge management, um, for task and project management. And so people are in it day in, day out. And so we have an ability, I think, to look at business, the business and drivers of the business and tie them back to user behavior and, and customer behavior in a way that I actually think very few companies in kind of B2B SaaS do. And I've worked in B2B SaaS before. And so I think that was something that got me excited initially. And so when I joined, really it's a mix of this kind of normal kind of business internal analytics use cases of data, which were getting quite significant for the reasons I just stated. And then in addition, some data products or some data— some products that depended on our data infrastructure that our team was supporting through building out and scaling data infrastructure for the company. So search, we were just talking about traditional search a moment ago. Search is a part of the product experience in Notion that we have to have because we have very large workspaces in a lot of cases. We launched features around workspace analytics for our enterprise customers last year so they understand how people are using Notion in their workplaces, et cetera. All of those depend on infrastructure that we were also actually using for internal analytics. As well. And we had to scale that infrastructure further for these use cases. And then AI kind of came along about, I want to say, I guess, 8 months ago now almost.

So we'll spend a good amount of time on AI. While we are on the topic of infrastructure, to the extent that you can talk about it, what does a stack look like? What do you guys use as a data warehouse, like the various tools and systems SIMs.

Yeah, for sure. Some of those, we've let people have logos of our logo on their websites. I feel like I can share those ones. I mean, we use Snowflake internally for the core data warehouse. We also have our own data lake infrastructure that we've built out ourselves on top of S3. We're in AWS, mostly in AWS today. We use for internal analytics stuff, we use Hex as a product actually quite a bit. So we're an earlier user of Hex. That actually happened before I even started. And we've expanded it quite rapidly across the company. We, you know, we have used a lot of, I think, you know, maybe I'll step back a little bit. In my understanding of the history of the data infrastructure at Notion is that it actually predates any data team at all. We had a head of growth who was there in probably 2018, 2019, who stood up pretty much exactly what I think you would stand up if you didn't have a data team and you wanted to use various vendors to solve your problems at that point in time. And it was, Snowflake, lots of companies use Snowflake, but we had a lot of other, some of these other vendors, I don't wanna name all their names, but things you use for ingestion, things you use for data transformation. And I think what's happened over the last couple of years at Notion is there are parts of our stack where there are parts of our business or product needs we wanna support where the scalability and cost effectiveness of those tools starts to not quite work for parts of your, stack. So eventually, we went out and built our own event pipelines. We went and set up our own way of doing— of landing data for our largest data sets in the data lake because it really was not scalable for us to use some of these vendors anymore. In other cases, there's long tails, I think, for a lot of companies of different integrations in terms of vendors other parts of our company are using that it makes sense that we don't want to really build that long tail ourselves. And so we've had to think about that. But what was interesting to me is as AI started to become more of a focus for us, some of this home-built internal infrastructure that we were already starting to develop for some of those use cases started to be something that actually helped us and kind of was a bridge to getting some, some AI product experiences out faster, as well as some that we're working on that I can't share too much detail on today.

All right, so let's talk about AI and For context, and maybe that's obvious to everyone here, but Notion made quite a splash when you released the AI features for a bunch of reasons, one of which is everybody's wondering whether you can build separate defensible startups and whether if you started with AI, could you create an AI-first Notion? But then Notion sort of squashed that by moving fast. And then precisely as a second point, that how quickly you did it, I think was very well noticed and very well received by the community. So congratulations on that. But to start from the top on AI, so what does Notion AI do?

Yeah. So we are in the early period of AI features in products like Notion. And so we were focused on getting something out into the world pretty quickly. Today, Notion AI, I would say, is, um, it, it can add, it, it can add value to anything you're trying to do around creating content in Notion. Um, what we end up finding, uh, over time is that a lot of the most intensely used use cases of, of the current Notion AI experience are around kind of summarization, improving content, One of the really standard ones that a lot of people use is generating action items from meeting notes. That's a thing that comes up a lot. Summarizing that content, we have some automated ways of like you can create a block at the top of a page and it will automatically update with a summary as that page changes. So imagine you're developing, you're working collaboratively with your team, with your team, you're building out. I think we often are doing or writing like a product requirements doc or something like that, you want a summary of that at the top so people can quickly glance and see, is this the thing I'm looking for right now? So those are some really common use cases. Another one that came up for us a lot is, and this one's like, this is, this was very much a commoditizable thing is, but nonetheless really valuable really quickly is we have a lot of larger enterprises and largish enterprises using Notion and many of them are multinational and Notion ourselves, we have field offices in all over the world. Dublin, Seoul, Tokyo today. And we find actually that translation, like automated translation is something that these models are quite good at really rapidly. And being able to have that capability in the documents that everyone at your company in different locations is working in makes a really big difference. So imagine we're launching something like Notion AI and we're going into the Japanese market with that. They were very excited about it, we could actually generate content really quickly to help get us— like, really take all of the things that our product marketing team is building and have the Japanese sales teams have access to that and then build off of that. So there's a lot of use cases like that as well that we've found really useful internally, and we've seen our customers using too.

So many questions here, but one—

I guess, how does it work?

Meaning, to the extent you can talk about it, read in the press that you guys use both OpenAI and Anthropic. I guess, what is the stack that enables you to do this?

Yeah. And this is evolving really rapidly. So I think we're taking a pretty pluralistic approach to how we're integrating with partners right now in this space. And while those are the two that I can say are, yes, publicly, like we have leveraged both of those partners, we are also designing things, I think, in a way where we know this space is changing really rapidly. Just heard earlier about this debate around open source models versus these really huge LLMs coming from some of these companies like Anthropic and OpenAI. And we want to be set up in a way where we know what kind of the core components of the stack that we're building are, but we can, in many cases, swap vendors out relatively easily if we need to because these things are changing. We're already working on some— We already have some ML on the kind of search side of the stack, so we can leverage some of that infrastructure that we've built as well. But I will say it's very early, and I think actually a lot of the decisions we're going to be making about Notion's AI stack are going to evolve a lot as some, like, new capabilities come out in the coming months and through this year. So I would say it's very early in terms of what we needed for, for that part of it. I want to go back to one thing I think you asked earlier that I didn't fully answer, I feel like, which was around Notion was very fast to market. How did that happen? One thing just to call out is while no one saw Notion AI capabilities before ChatGPT, I'm not going to give us too much credit. It wasn't way, way, way before, but we were definitely working on this a little bit before that. And there was a pretty big focus at that point. The other thing we did was Once we realized that this was something we needed to start really investing in, our approach was to actually bring a small group. It was kind of a tiger team type approach, like bring a small group across my data platform team, across some of our product engineering teams, a very small number of people actually, with a lot of targeted effort from our co-founders as well to get an initial product to market quickly. And so it was a small group. And my job and the job of a few of the other of us kind of in that management layer was really mostly about how do we unblock this group to just get this done and reprioritize things at the company so it can happen quickly. Now, over time, that's already evolved maybe once since then. And there's actually an AI core team that is part of my group now. That's going to evolve again as we continue to make these investments. I do think the right way to get to market quickly for us at least looks a lot different than probably how I expect AI development to operate at Notion in 3 months from now, 6 months from now.

So that's one key takeaway then, support from the very top, the founders, and then a small group just to play it back, a small dedicated group. OK. What do you think companies need to already have in order to be able to do that, whether that's a team or platform or having your data in order or what are the preconditions to being able to successfully deploy AI quickly?

Yeah, I think it's going to look pretty different depending on what stage your company is at. I think for us, the biggest things that we benefited, are, are benefiting from, I think are, um, number one, that a lot of the initial use cases for, uh, around Notion AI were actually things that we could build basically by integrating some partners, uh, by leveraging some of the partners, uh, that, that you mentioned. And, um, just like, uh, AI-aware product engineers, which we happen to have a few of. So that was great. And it also helps that one of our co-founders who still actively writes code was in his own world working on this for some time as well. And so that gave us an advantage initially. Now for like where we're going from here, I think for us at a company at our stage, it's really, it's beneficial for us that there were some critical data transformation and processing problems that we were already working on. And that's an advantage for us, I hope, going forward. So for example, some of the ways we need to process all of the content in Notion workspaces, given the way that data is structured, depends on some really large-scale processing that we had started to do for the purposes of, to some extent, internal analytics for search and other use cases as well. And so now we're kind of in this mode where, um, and this is kind of fun 'cause I feel like I can talk about the tech without telling you actually what we're building.

The—

we're in this fun, fun place where my data platform team, the folks, some of the folks over there are working on, okay, how do we scale that processing to support some of these AI use cases that are going to be quite intense? And then we'll be looking at new partners and, and other adding new parts to our stack on the AI side. So one of the things we did a few months ago was actually move some of my data platform team over to be part of this AI group. Because they knew that infrastructure already and could build on top of it. Meanwhile, the data platform side is thinking, how do I scale all of this up to work in a way that has the latency requirements and real-time-ness and all of these things that we're going to need for some of the things we're going to be building and rolling out soon?

What about the UI/UX aspect of this and how to just deploy AI in production at scale to a bunch of users who may not be familiar with AI, what AI can do?

Yeah, for sure. So first off, I think the initial version of Notion AI that you all have seen is, it's not a radical transformation of the Notion user experience. I think that's on purpose. I think we're all gonna be going on a journey in the next couple of years in, and this I think is a big part of what people talk about when they talk about a platform shift or the term, the buzzword that we keep hearing.

Is the—

we need to start from a standpoint where people have an understanding of how they start to use this product. Over time, I actually think there will be a more radical transformation to the user experience. I don't think it's all— I personally don't think it's all chat all the time is like the way we are all going to interact with all products. But there are going to be— we're already starting to think about, OK, as we go deeper in this direction, what are some of the ways our overall product surfaces are going to need to change and evolve? Given the overall mission that we have around making software toolmaking ubiquitous, around allowing really easy collaboration and knowledge sharing, I do think that's going to change over time. But in the short term, it's really about how do you create ways of integrating AI into the experience in ways that are very natural. And I think Notion's advantage here as a company and as an organization is that design has been very central to our experience from the very beginning. So I'm gonna correct one thing you mentioned at the very beginning. So Notion, nobody really used Notion in the world until 2016, but the Notion actually started in 2013 and, um, and really was more of almost like a research project for a period of time. Um, and the Notion 1.0 release that came out in 2016 that I think started with people using Notion for docs and wikis in 2018 for databases. That really stemmed from a period of time in which Ivan and Simon, who are two of our co-founders, were basically restarting this company, living in Kyoto. You can go read about more about this online. You don't need to hear it all from me, but essentially it was the two of them going back and forth for like 16 hours a day. And it was one of them working in Figma doing wireframes and mocks, and then the other one implementing it and then them playing around with it and repeating over and over again. And with Notion AI, when we were starting on that, we brought design in very early too. So part of that team, one of our earliest designers started working on that team. And I was like, you know, watching this in Slack almost like I wasn't that in the weeds of that part of the product development process. But, you know, really like day by day, like here's some mocks, getting feedback from all the way up to the co-founders, getting feedback from them on it, doing another rev on it, testing it out. And then the last thing I'll say is we use Notion so intensely internally but then we dogfood the crap out of it ourselves. And so, you know, we were obviously playing around with this for quite, quite some time. And that's why if any of you were in the alpha before it fully— Notion AI was available to everyone, which is for a couple of months, there were things that we added in over that period of time that we were testing internally constantly. And so I think that makes a huge difference for us as well.

So not an overnight success, huh? What do you find exciting about where this whole space of generative AI is going in terms of, you know, what do you think the cool products, companies are versus what may be overblown as of now?

Yeah, that's a great question. I am excited about agents, which is the thing that came up recently. And I'm excited about that development. And it's definitely something that Keeping in mind, and there's ideas around. I think we're excited about, for us, and this is one of the things that I think is really exciting, what we want to be able to do today, if you use Notion AI, you are generally just the generative things that it will build for you are built off of the content, basically on the page that you're working on in Notion. And we see the opportunities around AI as being much broader than that, around really, again, how do you share and transmit knowledge across a workspace in Notion. And I think, as I think more broadly, I think we're going to see a real transformation in the way that knowledge work is done. And there are tedious aspects of this that we have to do today ourselves that, um, we think AI can help a lot with. So one thing I can share here is that, you know, we are, um, excited about some, some major improvements and kind of an all— a much more cohesive experience around how you manage projects in Notion. Um, and some of that's going to be coming very soon, and it's going to be very much I would argue the first AI-enabled experience of how to do project and task management. And I think there's some really exciting things we're going to be able to do with that. It's going to depend on some stack and infrastructure that we haven't, that we haven't shared yet. But it's a really exciting area for us. And I think it's a big part of the value proposition of what makes Notion different, which is unique, which is that all of this is happening in one place. And maybe the kind of like philosophical thought I'll leave you with there is like You know, one of the most interesting use, uh, most transformative use cases I think of generative AI and large language models is the ability to bridge between structured and unstructured information much more easily. And that's like right in our wheelhouse because Notion ideally is a place where you both are storing a lot of structured information and unstructured information. In fact, the mere concept of a Notion database was the idea, uh, and the kind of innovation of it in part was you have structured information in terms of the fields in that row of the database. But also each item in that database is also a page that can have tons of unstructured content in it. And so I think there's going to be some really interesting things we're going to be able to do there. It is going to depend on some of the capabilities from some of the vendor— the types of vendors and companies that we were just— that spoke earlier today. I can't say I'm working with them specifically, but those technologies are all relevant for that process and that development.

All right, very cool. Let me turn to you guys for questions.

Thanks, Daniel Sternberg. You might recall Evernote got some negative attention due to them changing their terms of service to let employees, data scientists look at data for their modeling. How have you thought about the privacy and proprietary information, and especially if you're trying to use third parties as well for some of your AI functionality?

Yeah, that's a great question. So, and the privacy and security side of this is a really important one, and it's something we take very seriously. And it's also something that I think this entire industry is grappling with right now as well. So for the first part of this, like, are internal people at Notion looking at your data for these use cases? The answer to that question is just no. We, unless you're opting into specific things we can do with logging. So we do not, we do not actually like look at pieces of content from external users' workspaces. And the good news for us is actually We were able to go a long way often in testing out new, new, uh, new AI capabilities with just internal, uh, with our own internal Notion environment. So we do that quite a bit. In terms of the third-party part, it's a great question. So first off today, um, a, we users, uh, and enterprise customers do this all the time. Um, one can completely opt out of that experience altogether. To— we have agreements with our partners that they were not using that data to train models, etc. And you can turn off Notion AI for an entire workspace if you want to do that. To your point, I think this is one of these areas where we're going to be figuring out in this industry in the next year, in the next few years, and probably even sooner. Obviously there's advantages to open source models in the sense where obviously we stood those up ourselves. Now you're not thinking about subprocessors and other third parties having access to that information. So it's definitely something we are focused on. Um, our legal team is focused on, um, and we want to be transparent about where your data is going. And so as, uh, today we have a limited number of providers we're working with and we are only sending content to them when you are choosing to use Notion AI. As we start to see use cases potentially that use, you know, are using, sending more data from a user's workspace, from a company or a user's workspace to these third-party providers, obviously we wanna have a way for people to make a decision about whether they want that.

Hi. So you mentioned practical applications. Just a fun thought exercise for you. If there was one thing that you could build practical-wise with generative AI, what would it be?

There's one thing I could build for Notion or just generally?

In general, you personally.

Oh, that's a really, that's an interesting question. Um, trying to think of what comes to mind first. Um, I think, I think for me, Personally, I would love to have a way— I have two young kids. So there's some examples for me that come to mind with my kids where I would love to be able to create more personalization around how they learn. So personalized learning, I think, is something I'm excited about. I'm very curious about how this is going to transform the education space because I have some kids at various ages entering entering school. And so that's one personal use case that I'm, that I'm interested in. I think for Notion more generally, I'm really excited about the ways that AI— I think it, you know, I think of it as is going to make, is going to make the development of ways that we collaborate, knowledge inside of our Notion workspaces, is going to kind of take away a lot of the tedious parts of doing that and really focus on humans doing the parts that humans do best. So for example, what's really important in a team is how do we work together collaboratively? How do we resolve conflict? How do we build— how do we make progress together? And there are a lot of things we have to do along the way there that we have to jump in and work on ourselves. That are very— are quite tedious. And I think AI can help a lot with a lot of those use cases and help us focus more on those human parts as well.

All right. That feels like a great place to leave it. Thank you so much, Daniel Sternberg, for sharing all of this. Really appreciate it.

Yeah.

Thanks for having me.