A recorded conversation (YouTube) between Steve Huffman and Alexis Ohanian about "data debt": the risks of not instrumenting products early and practical advice for minimal viable data collection. Speakers share anecdotes (difficulty measuring historical user counts, early Mixpanel failures) and concrete recommendations (log top user actions, keep historical baselines, follow best practices for event collection).
Questions this video answers
"It's probably somewhere between 250 and 320."
"Pick your top 5 to 7 most important user actions."
"Only like 6 months old."
Chapters
We just didn't. We didn't know our traffic for years. We didn't measure anything. We. The product development by intuition for a very long time. Company ended up working out. I'm still not quite sure exactly how many users we have, and it's a huge, huge pain in the ass now how many users you have. When you get to 300 million, it's like, you know.
No, in all seriousness, we have no idea how many unique people use the metric.
It's probably somewhere between 250 and 320.
Anyway, at Hipmunk, we're actually really good.
About it from day one. And we were really, because we didn't have that kind of instant feedback with the users. We actually got really disciplined about testing and analytics and measuring and watching. And that company lasted, honestly, quite a bit longer than I think it would have without doing that. And so if I could do it again. There are a lot of corners you can cut when you're building your mvp. You can create all sorts of tech sins and corporate sins and, I don't know, be completely dysfunctional. Users don't care. But data debt can haunt you because you can't make up missing data and eventually your intuition is going to fail you. So I would think very carefully about what's the minimum viable data. You don't even have to look at it, just log it somewhere so it's.
There when you need it to reinforce that point. Having historical baselines for important user actions. You will thank yourself later. Every time I've ever been like, oh, right, how many people are using that feature? And we don't know and we can't find out. And now, even if we find out how much it is now, we don't know if that's up a lot or down a lot. And we have no idea what the trends are. It's awful. And you can't make a decision for another three to six months as you build that baseline. Actually, I got the best advice about this from Suhail, who runs Mixpanel, when he was convincing us to try Mixpanel for the first time, which, by the way, didn't work because Mixpanel at that time was only like 6 months old. And it broke when we tried to instrument everything. But it was interesting to me to.
Use Mixpanel before you came to that conclusion.
Yeah, you're welcome. It works better now. It works a lot better now. We went back to it. We're still on Mixpanel today. We use a bunch of other stuff too, because we're a little too big for just Mixpanel, but Mixpanel, the advice Suhail gave me that was, I think more important than the software or using Mixpanel specific, even though I do think it's good software, was pick your top 5 to 7 most important user actions, whatever those are, and just log those. You don't need to log everything. You really don't. Most of the things your people can do with your product aren't important. There's maybe five things people can do. Like for Reddit it might be like vote on something, comment on something, submit a story, load a page, I don't know, some short list of things. And those are important. And you can kind of just ignore everything else. If it's not instrumented, who cares? And so we instrumented. I watched a minute of video, I sent a chat message, I followed a stream and I bought a subscription. And that turned out to just give those four actions gave us a huge amount of insight into what was going on with our customers that we hadn't had before.
This is one of those areas where if I could go back and give myself just advice for like 30 minutes that would like dramatically affect the trajectory of a company, it would be around this topic. And so if I were in your shoes, I would just take half a day and just read Best practices for collecting events and storing them.
Another thing I'd recommend is use a third party service, collect logs. By all means, put them into your own system because I guarantee you you'll want to do something the third party service doesn't allow at some point. And you'll be super annoyed if your data is only in the third party service. So collect logs or whatever, but just in the beginning, you don't have time to mess around with building analytics tools. So just use Mixpanel or use Google Analytics or use whatever, anything. They're all better than what you need right now. Later on you'll need something more complicated. You don't need it today.
