Data Doesn’t Have to be Hard: Three Data Myths and How to Bust Them | Mailchimp's ex-Head of Data

Welcome back to Workshop Wednesday. Whether this is your first or your 40th, we are so excited to have you and have you from all over. Thanks for putting in the chat where you are coming in from. Every Wednesday, rain or shine, we are here at 10 AM Pacific to make sure we make this a valuable hour or so of your time to share the stats, the data, the info you guys need to be successful. With that, if you do have questions for this session, make sure you get them in the chat early. I will be messaging you privately to ask if you want to say your question out loud at the end of the session. And with that, I will start introducing who our session is actually about today. It is John Humphrey. Thank you so much for joining. He's former head of data platform product at Mailchimp and the current principal at mFact. He will be discussing Data Doesn't Have to Be Hard: 3 Data Myths and How to Bust Them. Welcome, John.

Thank you, Kaitlyn. So great to be here and so excited to get to chat with everybody and who's all over the, the globe today.

Yeah, John, you're doing a lot for us this year. You're at Europa, your workshop, and you're going to be at Annual too.

Yes, I'm very excited about being at Annual in a couple weeks. And yeah, it was a great opportunity to get to connect with so many amazing startup founders at London. And so just couldn't miss the opportunity to be back in front of such an amazing group of people, you know, today and in a couple of weeks in, you know, the Sorare annual event.

Yeah, we're so excited to have you, and thank you again so much for joining us on Workshop Wednesday. I have a few questions to get you— to get to know you better, even though if you were at Europa, you might know you already. Are you ready to get started?

Let's get started.

All right. So with that, how would you define your day-to-day role?

That's a great question. You know, you could define it in a lot of ways, you know, you know, probably in kind of the HR systems, you know, at Mailchimp, they would define me as a product manager. You know, people would often define me as, you know, a data analytics or data science person. I really think about my job at its core is how do you sort of unlock data at scale and activate it in order to drive better outcomes for your customer. That's really the thing that I care about, whether that's done by building a dashboard, whether that's done by building a new product. I'm kind of agnostic as to the how. It's really the what that matters. And so as I— again, as I look over the arc of my career, that's the thing that every piece of work I have has in common. And whether that is— Explaining to folks what data can do for them or getting hands-on and making that actually happen through code or through defining metrics or whatever. That's kind of, you know, what I do. And so while the specific tasks I perform in a day may vary wildly, that theme has been very evergreen through it all.

Yeah, I think that's a good reminder that no matter what the title or the position or whatever, it boils down to what are you improving and what is that like value add?

Exactly.

What project are you working on today you are most excited about?

Yeah, the project I'm working on that I'm most excited about, as you're alluding to, I was, until very recently, I was at Mailchimp working on a bunch of the data platforms and external-facing products that we were building for our customers there. I'm now working on a little project that I'm calling MFact, which is short for Move Fast and Count Things. And it's really all about how do I help startups unlock the value of their data. As we'll talk about in a little bit, I've been everything from data hire number 1 at Goodreads to the guy who signed off on all the non-GAAP accounting metrics in an S-1 to get LegalZoom public. And I've even been at public companies far bigger in scale than any of those things. And the thing that you find as you go from place to place is the problems are more like than different. And so really just trying to kind of break through some of the FUD and FOMO that people have around data and get them moving so that, you know, they too can do the exact same sort of things I was talking about earlier in terms of using data to really sort of activate and excite their customers at scale. You know, just figuring out how do I sort of help others do more of that is the project I'm most excited about these days.

Yeah, congratulations on the shift. And that was a new fact for me, what MFACT stood for. So thank you for sharing.

Mm-hmm.

In the last 12 months, what's the hardest thing worked on?

Ah, yes. So I'd say the hardest thing I worked on probably, you know, most recently was, you know, coming into the sort of product role that I was in at Mailchimp, where really what we were focused on is how do you sort of, you know, build essentially what we were characterizing as a, you know, CDP or customer data platform in order to really help, you know, Mailchimp's customers activate their data at scale. You know, kind of coming into a product that is, you know, a little over 2 decades old, and reimagining how it uses data from the ground up is actually a really hard thing to do, both from a technical perspective and also just from kind of a mental model perspective. And so really being able to come in and sort of, you know, get some sort of vision around, okay, you know, what does it mean to, you know, bring data into the product as a more native element, and then, you know, really unlock it in ways that, you know, makes life better for, you know, Mailchimp's customers and the customers of Mailchimp's customers. That was a really hard and interesting challenge, not just from a data and technical perspective. But as someone who's honestly never product managed before, just getting in there and figuring out like what is actually hard about product management was a really useful experience. And, you know, even though now I'm shifting more back to my roots of being, you know, a data guy first and foremost, having walked a mile in the shoes of a product leader and somebody who's responsible for a product roadmap, I look at that job differently, even though I've worked very closely with PMs for years. So that was a very, you know, sort of challenging and an exciting thing. You know, we were able to make some good progress in terms of kind of setting the vision for that. But, you know, it really was a great learning experience for me as well as to, you know, what is it that makes the PM's job hard? Like, no, really, not, you know, like walking a mile in their shoes. So yeah, that would be kind of the thing I would highlight from like the, you know, the last year or so.

Yeah. So data doesn't have to be hard, but project management is, is the moral of the story.

I think the truth is nothing has to be hard, but things invariably become hard. And understanding why they become hard then is the beginning to going down a path and where we start to disentangle some of that complexity and make this all a little more approachable. And it doesn't matter whether you're a PM or a data person. Those avenues are always available to you.

Yeah, I love that. Well, with that, we will shift over to your presentation on Data Doesn't Have to Be Hard: 3 Data Myths and how to bust them. Again, thank you for joining, and take it away.

Thanks, Caitlin. So yeah, so really the hope of what I'm going to do with these slides here, which we'll go through in about 10 or 15 minutes, is really to be a little bit of cannon fodder and kind of source of inspiration for the Q&A. The way I think about it is that when you're in these sort of earlier stage sort of companies and dealing with some of these problems for the first time, my vague hand-wavy PowerPoint isn't the thing that helps you. It's the very tangible Q&A of like, I'm dealing with this specific situation. What do I actually do? About it, that's the part that's helpful. But I think at least in terms of giving people some sort of guardrails as to what is useful to understand about this space and what can give you some inspiration for Q&A, that's why I want to run through a few slides real quickly and just give you some ideas as to some of the reasons why I see people getting stuck on this front and how you can potentially move forward. Let's see. So yes, so a quick little bit about myself. I know we've already gone through some of this, but I will hit on a few things that we maybe didn't talk about. I've led data orgs a number of places. My most recent data leadership role was Chief Data Officer at Mailchimp. But prior to that, I was literally data hire number one at Goodreads back in 2012, back when we were a Series A company, even prior to the acquisition by Amazon and all the due diligence work that went into that. I've also led data orgs at large companies like Intuit, leading data for all the QuickBooks Online products. I've worked on the ads business at Meta. And I'm here to tell you, it doesn't matter how big or small you are. Your data are messy. You have hard data challenges. And so don't feel like, you know, that you're in some disadvantaged position as such. You know, one particular highlight of my career is, you know, once upon a time, I got to stand on the stage at NASDAQ as we took LegalZoom public back in 2021. I was brought in as we were kind of closing in on, you know, a public filing and basically brought together the data org, you know, in the first year, probably doubled the size of the thing and, you know, got got an S-1 out the door. And there was a lot that I learned from that experience that I think is very highly transferable into the sorts of, you know, challenges that many startups are facing today. And having sort of seen that arc from Series A to IPO, you start to recognize that there's some patterns that make this all a little bit more approachable. And, you know, like we've talked about, you know, nowadays what I'm really focused on is how do I help startups and startup founders, you know, get the most value out of their data. So with all of that said, I'd be remiss if I didn't acknowledge the elephant in the room that really permeates any conversation about data and how you get value out of it. And that is just that there's so much hype. There are so many buzzwords, so much jargon, so much sizzle, and at times it just feels like so little steak. And really, this is what gets in the way of people getting value out of this stuff more than anything else. And so hopefully today what we'll do is we'll get past some of the buzzwords and some of the jargon and some of the hype and get down to actual substance and how do you help use this stuff to help move your business in the right direction and better serve your customers. You know, as I sort of look at the conversations that I have with folks that are in your shoes, and as I've kind of been parts of organizations as they go through this part of the journey, I really think that there are probably 3 myths in particular that kind of stop forward progress. I'll kind of tee them up here real quickly, and then we'll dive into them a little further. The first one is that this stuff is hard, that because it's technical, because it involves this stuff called data, and that involves some mathy type stuff, that it just becomes unapproachable, and we should just forget about it altogether. Another one is this kind of myth of all or nothing, that if you don't do this to perfection, if your math isn't right to 6 decimal places, just pack up and go home now. Also a complete myth. And then last but not least is this sort of myth of scarcity, that you lack the resources, the people, the whatever it is to extract value from the stuff. This too is not— is simply not true. Let's dive into the first myth here, which is that this stuff is hard. And I would say whether you're talking about data, whether you're talking about the business in your industry that you work in, or any subject in life, anything can be hard when you don't know where to start. It has taken me years and years to learn everything I know about data. But you don't necessarily need to replicate that trick in order to get value out of this stuff. Really, what you need to do is take advantage of the fact that, in this case, you actually know where to start. You have a business, you at least have some traction. And with that, you have some opportunities that you're looking to harness, or you maybe have some problems you're looking to solve. And so start with what you know there and use that to narrow your focus. And if I were to have sort of one particular kind of takeaway that I'd offer up here, it's that actually business expertise is more important than data expertise. Knowing your business, knowing your domain, Knowing the problems that you're trying to solve for your customer is the thing that you and only you can know. Once you get it down to the point where you have started to focus on that a little bit, then to the extent that data expertise might come in handy, there's all sorts of ways to source that. There are sort of courses that you can take. There are people that you can hire. There are consultancies and there are products that you can sort of bring in as well. All of this stuff is available, but you and only you have the expertise in your business and your customer. And in truth be told, that's actually the most important thing all of this. Let's bust this myth even further though. One of the big pieces of advice I'd give here is don't invent problems. It's very tempting to look at whatever latest news is coming out about generative AI and say, oh my gosh, I need to go find a generative AI problem in my business space and go solve that. No, don't do that to yourself. You already know what the most important problems are to solve for your customer and make data and the techniques associated with them subservient to the problem, not the other way around. Similarly, I'd also say don't invent solutions unless you truly must. And I think what you'll find very quickly is that there are fewer places where you actually have to invent a solution. Let me give you an example from my own experience. So I was mentioning that, you know, I was part of the team at LegalZoom that filed the S-1 and took that company public. Uh, and one of the big things that I learned from that experience is that oftentimes as a startup, you're thinking to yourself, my gosh, what are the metrics that I want to measure for this business? How will I define success? How will I talk about this to prospective investors? How will I later talk to the street about this? What should I be measuring in day in and day out, et cetera? And you think, my goodness, like the possibilities are endless. The possibilities are not endless. The odds are that you are not the first company to file an S-1 and go public. You're probably not even the first company to do this in your space or with your valprop. Go read the prior art. Go look at some S-1s of companies that have come before you. Because you can safely assume that the investors who will someday put, you know, money into your company are going to have done the same, and they have some ideas based on that as to how they want to understand your company and the questions that they're going to ask. And so while here you could totally invent a bunch of metrics from scratch, the reality is in many cases they've been invented for you. And so leverage that and you— and sort of just, you know, course correct where needed. And so I think, you know, that's just one example, but there are plenty of places where the same idea holds true, that like look outward for, you know, existing art and leverage it wherever you can. And I'd say the other thing here is just kind of part of how you sort of deal with this is prioritization. Again, if the biggest problem that you're facing is that when you don't know where to start, everything is hard, take advantage of the fact that you know where to start and prioritize your investments in data no differently than you would prioritize any other investment you're making in your business. Everything you know about running a business didn't suddenly become null and void just because data got involved. Moving on to the next myth. Is kind of this idea of all or nothing, that again, if you don't do something to perfection, it isn't worth doing. And that's simply, you know, not the case. It really is about improvement, not perfection. If you can even get a couple, you know, points of improvement in conversion or retention or some such metric, that's real money that goes to the bank. Even if maybe you didn't do the most like methodologically pure analysis that's ever been done, it's okay. And, you know, I think another thing, again, sort of drawing on concepts that are very familiar to you in other contexts of your business. You know, if you think about like in product terms, there's the notion of, you know, kind of your minimal viable product or MVP. The same idea applies here. You know, if you're building a dashboard of, you know, metrics for your company, yeah, of course it'd be amazing to have real-time metrics that are in an automated dashboard that are accessible at a moment's notice. But you know what, sometimes just, you know, having somebody copy and paste some stuff into an Excel file that you email out every morning isn't half bad either. And so don't overlook, you know, good enough on the, on the road to perfection. And, you know, one example I often like to give when particularly sort of poking at this myth is if you've used an app called Strava that often, like, people who are into running or cycling use, I think this is a really good example of where, like, an application sort of finds a sort of medium use of, in the middle use of data. And so, you know, I've kind of shown an example here from sort of my own feed. And while you see at the bottom there's this little orange button that says View Analysis, I'm here to tell you, even as a data guy, I almost never click on that button. When I come to this page, I'm really looking at two things. I'm looking at average power, which you could kind of think of as like my input metric, and then average speed, which is really kind of my output metric. How efficiently was I pedaling the bike, and to what extent did that translate into the outcome that I wanted? And the reason I sort of drop this sort of example here is in many ways this isn't really different from what you are encountering as a leader trying to use data in order to run your business. Um, you don't need the analysis to be completely done for you. Uh, in much the same way that I can look at these numbers and I can say, oh, okay, well, I know that this was a particularly hard ride. And so actually 184 watts was good. Or I can remember to myself, oh yeah, like, you know, I, you know, had particular, you know, like I didn't sleep particularly well that night. And so then I wasn't riding as effectively in the morning. Then I can sort of contextualize and do my own analysis. And I don't need, you know, a bunch of analysts and a bunch of work done in order to try to make heads or tails of this. And so I think the moral of the story in any product that you might be shipping that has data elements to it, or the internal sort of reporting and metrics that you're building for yourself, is to remember that, you know, sometimes you just need a number to even get started. And so again, don't block making some progress on the idea that, oh well, if I don't completely provide you insights and all the knowledge as to what to do with this number, that then I haven't accomplished anything for you. That, that's nonsense. Really just focus on doing more than what you did yesterday. And then sort of the last myth that I want to talk about is this myth of scarcity, that you lack the resources, whether it be in terms of people or platforms or data, in order to do something better with data for your business. And this is kind of one of those funny moments in which I remind you all that as founders of startups and founding teams of startups, that you are experts in dealing with exactly this sort of scarcity problem. This is no different than any other context in which you encounter this problem. And so leverage the strengths you already have as startup leaders in order to figure out how to get value out of data, as opposed to kind of like throwing up your hands and saying, oh, I can't do this, this is simply unapproachable. And so I will kind of, you know, talk through a couple of points here, and then I actually have an example from my own experience that I'll close with as to kind of what this can look like. And as I sort of bust this myth further, there's really kind of two things I'd highlight for you. One is that by virtue of having a product with a database behind it, you probably already have much of the data that you need in order to get sort of the core metrics to understand your business. You know, that little, you know, sort of production data store that's sitting behind the application, like, has to know a lot of what's going on with your customers and your product, or else it won't work the next time a customer comes back. And so don't overlook this source. In many cases, yeah, it requires some transformation. You know, it isn't just going to be like out of the box perfectly the metrics you need, but at least you have the Legos with which you can build anything at all to start with. And so don't overlook the resources like that that you already have. And the other thing to keep in mind, again, sort of building on this theme of you don't have to be perfect, is also in most cases, we're not running clinical trials here. If we get it wrong, no one dies. And so don't be afraid to get in there and try stuff. And yeah, sometimes you're going to get it wrong. Just do better next time. But yeah, I think oftentimes people are afraid to do something because they think they need the resources to do it with great precision. And really, no, you just need the resources to do it all. The perfect example, you know, thinking back to some stuff that I kind of shared at, at Europa was, you know, oftentimes people don't run A/B tests because they're like, oh, I'm so small, you know, to run an A/B test to like 95% confidence will take months. Who says you have to run it to 95% confidence? 80% confidence is probably a lot better than what you're starting with. And so just recognize that there are these opportunities where once you sort of give yourself permission to be imperfect, just look for improvement, you can do more than you're, you know, doing today. And so one example that I like to talk about in terms of sort of making this all real, and then we'll dive into sort of Q&A, is when I first came into Goodreads back in 2012, and I was like the first data hire, and you know, there was kind of nothing in place. One of the things that they sort of came to me and asked was like, hey, you know, my boss was like, we need to measure user engagement and understand kind of growth in a scaled way that we can look at again and again. To which I, analyst having worked at larger companies, said, cool, let me just run analysis on the data warehouse. I'll build you a segmentation. I'll automate the reporting. In which my boss sort of politely informed me, what data warehouse? And you can do this all in Excel, right? And in that moment, all of like a day or two into my tenure as first time of being head of data at a 30-person company, I recognized that I was going to have to think about this a bit differently. And so to talk a little bit about what I did, you know, again, going back to the idea of oftentimes you already have some data sitting around, you know, the backend of Goodreads at that point was MySQL. There was a production replica that became my de facto data warehouse and I just made it work. Uh, another thing that I'd call out is that I build where I could, uh, to save money and then buy where I must. So for example, in terms of a lot of the transformation in order to define the segments, that was just me writing SQL and Python that, that cost you nothing other than the fact they were already paying me anyway. Uh, but then when it came to like how how do I get a dashboard out to the entire sort of company every day? In that case, we were at least able to pay for like a Tableau Server license so I could automate the distribution rather than me having to copy and paste in Excel together and then email it to like 30 other people. But again, the principle was start with building things where I could and then resort to pulling out the corporate card where I must. And then I think really the sort of thing I came to appreciate through this experience is that scrappy doesn't have to be crappy. You know, just because we were getting creative and, you know, doing some things in, you know, kind of brute force kind of ways doesn't mean that we produced a bad outcome. In fact, the segmentation that we built ended up being quite material to not only helping us understand our user base a bit better, but frankly, as we were going through the due diligence that then led to the acquisition, being able to tell a bit more compelling story as to who the Goodreads user was and what sort of things they did as a result of the platform. And so I kind of offer this up as an example of something that, you know, like, within a couple of weeks of me arriving at Goodreads as data hire number 1, this was something that we were able to do. And I suspect at all of your companies, there are similar such opportunities where, you know, in relatively short order, you can get something that you're like, that was really impactful and I had no idea that I could get there from here. But enough with the pep rally here. What you're probably wondering is how do I personally get started on this journey? And with that, I wanna sort of open it up to, you know, our Q&A session.

Great. Well, that is a wonderful segue to our first question, but I do have to say your Strava example. I love running, but Strava, the data kept telling me I'm a terrible runner, so I had to stop using it because of the data. So funny example. But our first question, um, A Michelle Lee, if you do want to ask it aloud, feel free. If not, I will hop in. Okay, I will ask it aloud. So her question was basically your last slide. Where does a startup begin? What data metrics are most important when it comes to an email outreach to potential customers slash clients? And how do you use them to improve your ROI?

That is a great question, Michelle. I would encourage to— I would say that's a great question number 1. I would probably personally start with question number 0, which is why are you writing outreach emails in the first place? What outcome are you trying to drive there? And do you have the right metrics in place to measure that? Because otherwise, what you will very quickly do is you'll regress to looking at your emails like click rate, unsubscribe rate, and like very sort of basic metrics that are speaking to the health of the email, but not necessarily to the business outcome that you're driving. And I've seen this all too often across, across my career that somebody comes along and says, hey, I had a great campaign, I got a whole bunch of people to like open the email and I had a relatively low unsubscribe rate. And I'm like, fantastic, did we make any money by doing this? And the answer was I don't know. And so I think it really does help to start with build a foundation of how do you measure your success? And then from there graduate to, you know, measuring sort of email performance. And in many ways, like measuring email performance, at least in terms of like, again, sort of core email metrics, those are super like tried and true and like battle tested, you know, like, yeah, open rate exists for a reason, click exists for a reason. Unsubscribe exists for a reason, but like it doesn't tell the full story. The story you care about is a specific outcome, and that outcome does typically sort of is sort of like unique to your business that, you know, for example, when I was doing a lot of the email campaign stuff with QuickBooks Self-Employed over at Intuit, the outcome metrics we were trying to drive there would be very different than, say, you know, the sort of metrics that we would have been trying to drive at Goodreads. So happy to like kind of take a follow-on question there. But I would certainly, I'd say like in some ways the email metrics come second. The like business outcome metrics that the emails exist to drive would be the place where I would go first.

Got it. If you do have a follow-up there, hey Michelle, please let me know. Our next question comes from David. Tips on sharing data learnings with teams, library sharing, and then he mentions it's not David's autopilot, it is David asking the question.

Yeah, I think that's a great question. And this is one where some of it comes down to a couple of things in my experience. And I don't think there's a single right answer for everyone. One part of it is how big is your team? But the other part of it is, frankly, how does your team share learnings on everything else? If you go and create some very inorganic bespoke means of sharing data, like you say, oh, well, I'm going to put everything in Jupyter Notebooks because that's what all the cool data kids do. But nobody in your organization knows what a Jupyter Notebook is or how to get to one, and it's a workflow that they would never otherwise go through, then that's probably— you're probably just sending data off into the abyss and learnings into the abyss, never to be seen again. And so really be thoughtful as to how does the rest of your work get done. And I would also say another couple of things I think about here is to the extent you're very early stage and you tend to be an engineering-heavy and engineering-led org, know that a lot of what we do in data, especially in 2023, follows engineering patterns. And so the idea that you might, you know, have like an internal wiki page or that you might put some of this stuff in Confluence or whatever sort of constructs you have for sharing engineering knowledge will work relatively well for, you know, an organization at that size and stage. Where I'd say that starts to break down a little bit, and I've, I've seen this in my own experience, is when you have, you know, for example, you know, there's a contingent of your organization that's very like kind of product and eng, and then you have more of like your marketing folks. You know, the example I give is when I tell the former, hey, the way to request work from data is go use Jira. They're like, cool, got it. When I tell marketing or operations, go use Jira, on a good day I get a blank stare, on a bad day I get a dirty look. And so that would be kind of the advice I'd give here is, as much as possible, leverage existing norms for sharing both data and learnings about how to work with data. But then to the extent that you need to invent something novel, certainly engineering principles in many cases apply. And yeah, Jupyter Notebooks are not a terrible thing. But again, know your organization, know whether that's a place that people are ready to go.

A follow-up question to that. I agree with you using what people are comfortable with is always the best method. What about how do you explain data and metrics to the sides of the business that are maybe not as familiar with these terms? How do you make it digestible?

Sure. You know, it's funny because that's like a big part of what I've done over the course of my career is deal with that sort of like what I affectionately refer to as English to English translation. And so, you know, it really becomes— again, I think that principle of don't invent solutions is really important. Your business, your non-technical business partners already have a way of talking about the business. And they already have problems that they're trying to solve and they're focused on. And by and large, to the extent that you can learn that language, and speak in those terms, and essentially be there to fill in blanks that they can't fill in yet and provide a little higher level of resolution in the conversations that they're having today, that tends to, you know, sort of pave the path and allow you to move a lot faster. You know, don't invent constructs, don't talk to them about R-squared of anything, avoid the buzzwords, but really talk about like, hey, you are, you know, customer operations. Let me use an example like at LegalZoom, you know, a big part of what goes on at LegalZoom is that they are essentially filing things such as helping you form an LLC in, say, California. And so there's a bunch of literal paperwork that happens at some level in order to get that out the door and get the customer what they need. And so the person who is responsible for making sure all of those forms get filled out properly did not come up as a data scientist, but they certainly know and care when they have a massive backlog and their turnaround times are rising as a result. And so if you can speak in that language and you're gonna talk about it like, hey, like, here's your backlog and here's why your backlog is getting bigger. It's because, you know, the time spent servicing a given request has gone up and volume is going up because marketing spent more money at the top of the funnel. Therefore, you should probably somewhat look for efficiency gains. Why is, why are things taking longer? But some of this might just be solved through you need to staff more because you didn't do anything wrong per se, just Marketing spent more money and brought you more work. When you can start to speak in those very tangible business terms that people are already speaking in terms of, that often sort of becomes the connective tissue. And frankly, it makes you better at leveraging data to good effect because invariably they will come back and say like, ah, yeah, that's great advice, but in practice I can't do anything about that. Here's why. And then the insight you gain from that, that context becomes very useful in terms of how you think about like what you might do next.

Yeah, that English to English translation definitely hit hard. I'm in agreement with that. Our next question comes from Jaco.

Hi, everyone. Hello, John.

Thank you.

Thank you for your time. So I've got a question. Interested to learn more about, you know, which metrics, if any, you believe are important to track for an early stage B2B SaaS startup which has sort of customer self-service or self-service onboarding capability. And I was just always wondering, at that point, do you start looking at more B2C type metrics? You know, almost thinking about it as an e-commerce type setup where you have bounce rates, basket abandonment as such, or how do you tie together the, the SaaS metrics and then the, the stuff that happens before that? You know, sort the funnel, I guess.

That's a great question. And in many ways, it's one that's a bit near and dear to my heart because as I think about my experience in particular at Intuit working on some of the different QuickBooks businesses, in particular QuickBooks Self-Employed, which on the surface looks like a B2B product, but in reality is very B2C. And I think the answer becomes you draw inspiration from both. And so, for example, if I think like you're— I think you're kind of the sort of breaking it up into the right pieces that, uh, for sort of the self-serve kind of onboarding, yeah, that's, that's a very sort of B2C place that you go looking for inspiration. Uh, but in terms of the problem, the business problem you're solving, in many cases that probably still follows more of a B2B pattern. And so, you know, so much of this is that, you know, borrow the cliché, like steal like an artist. And so it's knowing where to draw inspiration from. And I think you're spot on that for some of the onboarding stuff, you will need to draw inspiration from B2C. Now, the challenge you will run into most likely is B2C metrics and sort of B2C, like, application of data often works because C is a lot of Cs. Um, you know, Facebook is able to do what it does because it has 2 billion users, and so it can run an experiment to significance in probably the time it will take us to have this conversation today. You will need months, maybe quarters to perform the same trick. And so then draw inspiration where you can, but also recognize, yeah, like inherently one of the hard things about B2B is that you never get the high volumes of a B2C product even as you get more mature. Like I can even tell you, like from working on like the main QuickBooks Online product, it wasn't like we were running experiments in hours either. And so this is where sometimes it gets a little harder because, yeah, you just don't have the volume of data. But again, you know, to go back to an idea we were talking about earlier, don't let perfection be the enemy of good enough. It's sort of like when you're taking a multiple choice exam. As much as you would love me to come along and tell you that it's neither A nor B nor C, the answer is D. If I just tell you it's not A or B, that still helps as well. And so, yeah, I'd say in terms of metrics, yeah, I think you're right in the particular— like on the onboarding and customer service type stuff, it is going to be very B2C. But recognize that what will end up being hard is that you're not going to get the resolution or the fidelity that you want because the quantities are going to be thinner. And so just know that like your margin of error is going to be wider. I hope that helps.

Definitely. Thank you. Appreciate it.

Yeah.

Okay. Our next question comes from Harry. Which B2B data AI search tools are you most impressed with out there? Um, i.e., tools which which can give you AI searchable insights via chat-like integrations?

To be perfectly honest, I'm not there yet. Um, in terms of— I think we are in very early days, and I can still tell you that by and large, my workflow looks fundamentally the same as it looked about a year ago. Um, I would say if I look at where— if I spent my time diving into AI over that time horizon. We certainly have shipped some cool things at, at Mailchimp that were much more like in the sort of content generation kind of space. And I think for those sorts of use cases, like, yeah, there are places where a lot of what you see from an AI perspective is ready for prime time, as evidenced by the fact these products exist and they work. But I would say for the most part, as I think about how do I kind of do like data search, how do I start to, for lack of a better way to say it, automate away some of the boring parts of the, you know, the analyst job in terms of coming up with metrics and things of that nature. I think we're still more in the stage where the hype is more exciting than the actual substance of what gets delivered. Which then begs the question of, okay, well, how do we get there or get there faster? And I think the way that you get there faster is to not forget about the fundamentals. The better job you do of structuring your data now in terms of how you sort of capture it in your product, and then maybe even flow it into some beginnings of a data warehouse, the lower the bar will be for whatever AI tools come along to like make sense of that. Uh, because I can kind of tell you like where I have spent my time, like back in the day when I was a sort of, you know, IC analyst with, with hands on keyboard building dashboards, it wasn't in the like defining the metric work. It wasn't the like the like mathy insightful fun-sounding part. It was in the boring, dreadful part of how do I get the data from this table to join up with this table so that I have both the numerator and the denominator for conversion rate. Stuff like that ends up defining your existence, not the, oh, like, what's the right success metric here? You spend far less time in reality on that. So even as the technologies are probably, again, they're a bit more hype than substance at this point. The thing that you can be doing right now is, one, just getting your data better and better structured and making sure that you have everything. And then two, like, that grounding in business problems that we talked about earlier will make it clearer as these technologies move from being emergent to being more substantive, uh, which ones are helpful, are going to be helpful to you now.

I'm sure this is going to be a follow-up question. Do you have a data stack that you do use? Um, do you have any favorite tools?

I, I love this question, and I actually brought up this point, uh, during Europa, which is the amazing thing in the year 2023 is that there are so many tools out there that are good. Pick one and go. It's sort of like the existential question of, do I use BigQuery? Do I use Snowflake? Do I use Databricks? My answer is yes, use one of them. They're great. Um, are they all great in the exact same way? Are there Am I saying that there aren't pros and cons to one vis-à-vis the other? No, of course I'm not saying any of that. But like, as I think about where we were, say, not even a decade ago, the fundamental problem we had was that we had more data in our data warehouses than we had processing capacity to sort of deal with the volume and complexity. We have now hit a point where we've caught up. And frankly, most any tool you pick is good. Just so long as you pick one and get going. And so I know that's not necessarily going to win me any endorsement deals from Snowflake or Databricks or anyone, but the point being that all of this stuff has just gotten so much better. We've caught up. And so it's harder to make a mistake now. And even if you do make a mistake, it's relatively easy to recover from a platform perspective. So there's a lot of things I like out there right now. I'll tell you the only thing that I still in 2023 shake my head about is reporting platforms. It doesn't matter which one you pick— Looker, Tableau, you name it— they all are like materially frustrating in some way. And so when you start getting to that part of the journey and you're starting to like build out dashboards and things of that nature, like just make peace with the fact that while they are better than the things that we used to have, and they are certainly better than like you know, slamming all this stuff into Excel, they're always going to frustrate you in some capacity. The only good news I can offer is that when it comes to something that is an endpoint like a reporting layer, you don't have to pick just one. If you have a bunch of data warehouse layers in the middle, the chances that you will create like duplication and conflicts and confusion and, and chaos are pretty high. But at that sort of like very end of the line, if you will, If you want to have, you know, Tableau for certain things and you want to do NetSpring for others and Looker for others still, like, that's cool. Like, you can do that. Uh, and so just know that, yeah, all the platforms have gotten so much better the last couple years, but also know that dashboarding is still going to be frustrating. It just is.

Yeah, no, thank you. Um, do really appreciate it. If there are any follow-up questions on that, please let me know in the chat. Our next question comes from Anthony.

Good morning, John, and thank you guys for doing this. This is really informative. I appreciate it a lot. To that end, I am like probably a lot of people on your call, a single startup founder, so trying to, you know, take the time that we have and slice and dice and make everything work is hard. So my question is, Are you working with people? Can people work with you on a, like, a project-by-project basis?

Yes. And I don't want to turn this into, you know, John's advertising hour. But yeah, that's exactly the sort of thing.

Maybe just 45 seconds.

Yeah. I'd say that's exactly the sort of thing that I'm working on. So feel free to reach out to me directly.

How do I do that?

We can provide the sort of contact details at the end. Okay. Derrail for that for too long. But yeah, you know, the thesis that I've— the thing that I've sort of experienced over the years is that so often as an early-stage startup, you don't need an entire data function. You don't— like, you're not in a position where you're going to go hire a data scientist and an analyst and an engineer and so on and so forth. Like, it just doesn't practically make sense. And so then how do we sort of fractionalize some of this stuff and give you just the bits and pieces that you need? Like, that's in many ways really like the novel art that I'm trying to solve for. But yeah, again, I don't want turn this into like an advertisement per se, but yeah, we can follow up and provide contact details for you or anyone else that might want to chat with me. And certainly I'm easy enough to find on LinkedIn as well.

Excellent.

Thank you.

Yeah, of course. And a true, uh, SaaStr man avoiding his own plug. We really have trained him. There's no, uh, no salesy, no commercial. That's why he keeps coming back. All right. Um, we do have a question in the chat from David. If you do want to unmute yourself and ask this, I think it would be a lot easier if you ask it yourself. Um, let me know if that's a possibility.

Yeah, you can hear me? Yeah.

Yes.

Okay, wonderful. Uh, it was a little complex, I apologize. Thanks, thanks for this, I do appreciate it. Um, making the following assumption, please: a decent-sized startup, $10 million ARR, 70% year-over-year growth, multi-channel, so some self-service direct frictionless as much as possible, plus channel sales team on top of that sitting there. So the, the, the product is proven, things are working well, good product-market fit, but there's been some data gaps in development over time. And so I just love to quickly get your perspective on, um, the priority and sequencing of the following types of systems. No specific names or commercial details need to be involved.

Sure.

But something along the lines of a manageable, workable CRM. The one in place is not so wonderful. Nobody really loves it. Secondarily, um, perhaps a CDP that can involve sales and marketing, getting those connections. And then thirdly, perhaps something more robust, uh, a data— like data warehouse solution. Uh, and I guess this seems to be maybe the stuff that you're working on, uh, but I'd love to hear your perspective on those elements, please.

Great question. This one is also near and dear to my heart. Also today I learned that OHIO stands for Only Handle at Once, and I love that. I love that sort of acronym, if you will. That's very, very helpful. I would say the way I would think about your prioritization is I'm going to assume that each of the things you've listed here is in an equal state of, for lack of a nicer way to say it, disrepair. Are, uh, because clearly if like one of these things is relatively mature, even if it's the most strategically important, like that's, that's going to change the prioritization. But let's just— so let's assume they're all like an equal distance of repair. I would start with your data lake. The reason I would start with your data lake is because as you implement a CDP, as you try to do better with CRM, where's the ground truth for all of those platforms going to come from and, and sort flow into. It's going to be your data lake. And so like, get that layer well done, well, and a CDP and a CRM becomes so much easier. And in fact, I think one of the interesting, I think one of the interesting sort of trends in data, you know, I was talking about how platforms are getting a lot better, is that as data warehouses become more and more performant, they're becoming quasi-production systems. You may hear of some different vendors out there that are doing things in the composable space, whether it be composable CDP, You know, like RudderStack, for example, is kind of going down a path like this. Hightouch very much plays in this arena as well. And their whole thesis is basically we don't need to build a data silo over here and then have a data warehouse over there and never the two shall talk to each other. That again, your data warehouse is sufficiently performant that all of this stuff can like extract from and write to your warehouse. So get the warehouse right. Also getting the warehouse right. Forces you to think about your business. It forces you to think about what data are important, on sort of on what interval you capture them, how you transform them, etc. And all of those are decisions that you'll end up having to make for your CRM or CDP applications anyway. So you might as well deal with them now and deal with them in a clean and consistent way. This also frankly sort of mitigates some risk of vendor lock. Uh, if your data lake is in good repair, you can, you know, and some, you know, your CDP vendor suddenly starts to gouge you or something, You're, you're not like beholden to them. And so I'd say definitely start with the lake. Um, just real quickly for those who may be less familiar, the way I think about the difference between like a CRM and a CDP is a CRM really is more for like one-to-one sales motions. Like I have a person who's going to call up a customer, so typically larger kind of order size relationships. And a CDP is much more like one-to-many. So you, you're when I was mentioning earlier that at Mailchimp I was working on a, a CDP type product, you know, Mailchimp predominantly is focused on, you know, customers who are serving like many, many customers. And so then a CDP becomes a more automated solution. But those differences aside, they're in many ways trying to solve sort of the same problem, which is why again, they all sort of, you know, work their way back to the lake at some point. Does that answer the, the question?

Yes.

Very well. Thank you so much. That's very helpful. Well articulated too. Thanks.

Yeah. And thank you so much for asking the question yourself. You definitely got better outcomes doing it than if I had tried to encapsulate all that. So thank you. All right. Our next question. You mentioned going public and the data needed to go public. If you are going public, what data points should you be focusing on?

Short answer, it depends. Um, slightly longer answer, go read some S-1s. Um, like, there are companies, there are companies that have come before you in your industry, or at least with the same sort of thesis as to like, you know, this is why we are, you know, sort of disrupting our space. Um, and knowing how those stories were told is incredibly instructive. And now I will tell you that like, if you look at kind of the metrics in an S-1, they sort of fall into buckets. One is a whole bunch of like GAAP accounting metrics. I'm glad that there are people who care deeply about those sorts of things and know how to do that stuff. That's less my wheelhouse. But a lot of that's very, very formulaic. But then like all the sort of non-GAAP metrics that are really like the business performance stuff, like that's the thing where there's some level of art involved. And here's what I'll kind of tell you. So the process you will go through looks something like this. Obviously, first draw some inspiration from existing S-1s. And I'd also encourage you to do this not just if you're at the point where you're on the brink of filing, but frankly, the sooner you do it, the better, because the sooner you are measuring the things that you will be evaluated to when you do try to finally go public, the better outcomes you will get. Now, having said that, even with all that inspiration, what will still happen is this becomes a very iterative process. So you will have a first pass at metrics. Your bankers and your advisors will kind of look at that and say, that looks good, but that metric, like, that's not going to be flashy enough story. Can we like contort it 5 degrees this way? Or talk to me exactly how you define retention rate. And I'm here to tell you that there are infinite ways to define retention rate, and I probably used every one of them, uh, during the, the LegalZoom filing, uh, some of them twice. Uh, and so yeah, just know that it will be an iterative process, uh, but the more substantive conversations you're having with the, you know, the banking team that you're working with as to like, okay, so what is the story you're trying to tell and what furthers it, uh, the better. Uh, but yeah, just go and read the docs. But yeah, I'd say You know, for example, like subscription kind of, you know, SaaS businesses are going to have some metrics in common, like again, like conversion rate, you know, retention rate, retention rate measured both in terms of subscribers, i.e., customers and revenue dollars. Those will all matter. And so like you'll see some patterns, but it does really vary kind of from, from space to space. But, but again, the other big pro tip I'd give is even if you're very early in your journey, It's never too early to start doing this exercise.

Yeah, no, that is great feedback. Um, obviously this has been an interesting year in the VC world as well. What data metrics matter now that might not have mattered last year, the year before that, et cetera?

In many ways, I think this is the whole point, is that, and so it depends on how you wanna look at it. Either nothing has changed or we have gotten back to fundamentals. Um, I can tell you honestly, like, I feel like in many ways a lot of like the generative AI hype cycle and bubble has finally sort of regressed. Uh, and so putting AI every third word in your pitch deck isn't gonna particularly cut it. But like, just keep it focused on fundamentals. Keep it focused on the business problems that you are most important to your customer and metrics that show that you are in fact, you know, making traction and solving those things. Things. Just like a lot of what I was going through in the slides is kind of giving us as data professionals a bit of a kick in the pants and saying, hey, look, forget all the fun science projects, focus on solving people's problems. I think the same sort of guidance goes to founders and people on leading startups. It is much more of just a focus in fundamentals, whatever fundamentals may happen to mean in your space. But do I think some new metric has emerged or some new way of thinking about things has emerged. Like, no, I think we— again, I think we've gone back to, um, you know, really focusing on substance.

Yeah. Um, when it— you've mentioned now a bunch of times about using data to solve your problems. If you're going to that initial stage of what are your problems, what are some of the brainstorming you do there? What teams do you work with to really define those problems that you're going to have data solve?

It's interesting because I would say in many ways, the longer you can go in this conversation without uttering the word data, the better. Like, think about it as you have problems to solve and get really crisp on those, agnostic of whether you will solve it through data, whether you solve it through marketing, whether you solve it through engineering. Just really be crisp and like, what is the pain point you're trying to take out of your customer's life? What are you trying to make possible for them that isn't possible today? Only once you have done that, and you have done that in the way that you would do that— again, pretend data as a discipline does not exist for a moment, like just do it the same way you would normally— only then come back and start layering in and start thinking about like, okay, how might data be part of the solution to this problem? But again, I think the more pure and more dogmatic you can be about not letting data or not letting solutions creep in any earlier to the conversation than you have to. I think you'll get better outcomes.

Thank you. We have another question from David. I know you're getting your steps in, so I'll read this one out loud. Deep, but maybe interesting to others in the AI, if they have AI in their scope, level of protection a virtual private cloud offers in potentially sharing user data with OpenAI to draw new insights and inferences.

Yeah, this is one where I've worked a little bit with OpenAI in some of my sort of my previous role, if you will. I would not necessarily characterize myself as a deep expert in this particular topic. And so I'm going to kind of not touch the question. It does raise one point that I want to bring up, though, that I think is actually really interesting for people to understand. Oftentimes people think, oh, well, OpenAI has trained their model on this massive corpus of internet data. And if you look at some of the other models out there, again, trained on massive, massive corpuses of data. I, as, you know, relatively small startup with fairly limited customer base and small number, you know, and small amount of data, couldn't possibly begin to train such a model. Throw in the towel, give up, go home. Actually, no. One of the interesting things that is quite often true of a lot of the sort of algorithms that are out there right now is, for lack of a better way to say it, almost the extent to which you can kind of graft your little dataset on to an existing model with a much bigger dataset and actually still get really good outcomes specific to your space. So don't think that just because you have very limited data that you can't train AI to do some really useful things. Instead, think in terms of like, I can take a very large publicly available dataset and I can extend it with a very modest number of examples from my own base and start to train a model that way as well. And then of course, you know, we've managed to make it, you know, almost all the way through this conversation without someone uttering feature engineering or prompt engineering. And I'm going to sort of break the seal on that one a little bit and say that, yes, prompt engineering often is a viable workaround for these things as well. So again, I know in many cases we're relatively small companies dealing with relatively limited datasets. But the reality is that a lot of this stuff is actually surprisingly accessible to us, you know, even now.

Wow, cool. Any follow-up on that? Okay, well, with that, we are going to wrap up. Thank you, John, so much for joining us. All your in-depth answers to all of these questions I know has been really greatly appreciated. And thank you so much for these English to English translations that I'm sure have really helped everyone when they're thinking about data this year and in the future.

Thank you so much, Kaitlyn. As always, I really enjoy the opportunity to get to connect with, you know, the sort of, you know, SAS community, if you will. And hopefully we'll get to see many of you at Annual in a couple of weeks here.

Yeah, we hope to see you too. And if you do have questions about Annual or anything else Saastr related. Next week we have an AMA with Saastr CEO and founder Jason M. Lemkin. Looking forward to seeing you all there, and thank you again, John. Bye, everyone.