Inside Duolingo Unhinged Growth to a $500M Business | Cem Kansu (Duolingo)

We initially designed you to be a motivating coach within the app, but social media made it, like, this haunting person that if you don't do your lessons, he will haunt you down. That wasn't necessarily our creation, but we certainly, over time, leaned into it. Like, sometimes we made it, like, really wrinkly. Sometimes we made it, like, melty is what we called it. The face is melting. The last one we did is we made his face look like he got sick. Like, he had, like, boogers coming out of his nose. And one thing I would say we take pride in, we use the term unhinged as a very positive thing. Like, when someone says, oh, that change is unhinged. At Duolingo, that means you've done a good job. Like, you've really unhinged this change. Like, our app icon is one. When Duo does something really weird in our app, like, we're like, yeah, cool. That's really unhinged. And when I talk to other companies, I say, yeah. Like, one of our product philosophies is making things unhinged. People are like, is that a good thing? Like, do you want to make the product unhinged? But I think that's, in a way, our philosophy. I think it. It really builds character when you're building product by kind of breaking these sanitized product norms.

Well, welcome, everyone. My guest today is Jim Khonsu, head of product at Duolingo. Over eight years, Jim helped Duolingo grow to 100 million monthly active users and half a billion dollars in annual revenue. I'm super excited to talk to Jim about Duolingo's rapid growth and how they will use AI to democratize education. So welcome, Jim. Jim.

Hey. It's great to be here, Peter.

All right, so I grew up in China, and, like, trying to learn Mandarin is such a huge pain. Like, you gotta memorize all these words. But I. I love this quote from your CEO that Duolingo wants to make eating broccoli feel like eating ice cream. And it does feel like using Duolingo. It feels more like playing a game with a lot of rewards. So I guess just use some game terminology. Like, what do you think Duolingo's core loop is to make learning fun?

I think our core loop at the end is the Duolingo lesson. So if you look at Duolingo's homepage, we follow a linear path, similar to how a lot of games have been designed, actually, and the linear path of completing a lesson. And at the end of lesson, there's a lot of celebration. So there's various parts that become part of your core loop. When you complete a lesson, you extend your daily streak is a core part of the Duolingo retention mechanic. Another one is you complete it. Generally you start completing quests. We give you a set of daily quests to complete. That's another part of the core loop. But at the end, we are here to teach you a language or now we also teach math and music as well, actually. But when you complete a lesson, the reward is intrinsically what you came to Duolingo for, which is learning the topic that you came here for. And I think the quote you used is one of our, I would say, fundamental beliefs of building product. At Duolingo, we are certainly not the first or not necessarily the only product that teaches language. There's actually a lot of products that teach language, obviously. And if you don't have to get very technological either, like books will teach you a language. Like, books actually teach languages really well. But there's a reason no one learns languages just off of books, because it's boring. As. And I think the, the reality is if for us to succeed in teaching you a subject, it has to be fun. So this, like, this is kind of why, like, we want to make eating broccoli, which is learning something by yourself, which is hard to motivate yourself to do, feel like eating ice cream. And we do that through making the whole experience more fun, adding these game mechanics that are like the streak, the quests, the leaderboards, and adding a lot of character to the whole experience with our mascot duo or our characters. All of these are in the intent of making learning fun.

So let's talk about some of these features you've added over the years. I mean, first of all, the company is like, I think over a decade old and still growing 60% year over year, which doesn't happen to a lot of companies. Right. But if you look back at the eight years you've been there, like what have been some of the key products or features that have really helped drive this growth? Or is it just kind of a lot of experiments and updates?

I think it wasn't the case that one feature just changed all of our growth curves. It's certainly been over time as multiple features got introduced and improved our growth kind of kept compounding. That being said, there certainly are a few what we call growth levers that we've built over time that are known to help Duolingo's growth. One of them is the Streak. So the Streak itself existed ever since we launched Duolingo however, it's changed a lot and improved a lot over time. And to set the stage, the streak is exactly what you would imagine. It's the number of consecutive days that you have done a lesson on duolingo and it counts up the more you do it. And it has this natural retention mechanic. The higher you go, the less you want to stop using because you've already built a high streak. And the streak is a metric that a lot of our users take pride in. So it really builds longer term retention that compounds over time. Streak certainly has been powerful that over time we've improved it to be a lot more fun to use. So it's like it's a lot more animated now versus it was just a static image before. It's a lot more shown in your daily experience on your homepage, on your profile anyway. But the streak is one mechanic. The other one is leaderboards. It is just reality that if you see your name ranked on a list of other human names, you, the human nature compels you to not be at the bottom of that list. You want to be towards the top. And I think our leaderboards have been a big motivator for our learners to, well, just do more lessons. And that's how you win leaderboards. The more lessons you do, the more XP you collect and the higher XP you have. You can win your weekly leaderboard. It's a weekly cycle, but leaderboards have been a big growth lever for us as well. Third is push notifications. We are somewhat known at this point for having fully unhinged push notifications. We have a various set of notifications, but one of our notifications that gets memed a lot in social media is our practice reminders. So generally we send daily reminders because it really fits into the concept of learning. You really learn the concept that you're learning like language or math or music much better if you do daily practice. So we send daily reminders for our learners to grow that daily habit. And over time we realize if we write creative copy as written from the tone of Duo, our mascot, that performs better and that's a lot more delightful to receive rather than an app saying, hey, do your lesson. So our push notifications have become quite improved over time. It's a big lever for our growth. We've also improved various things like how we use emojis, the timing of notifications, personalizing them. Anyway, the list goes on. But the one that gets memed a lot is what we call the passive aggressive notification. If we send you a notification for A few days and you don't come back to the app. We don't keep obviously spamming you forever. We stop at some point and the last notification that you receive is meant to be like a text you receive from a significant other that is kind of pissed off. It reads, hi, it's Duo. These reminders don't seem to be working. We'll stop sending them for now. And this has taken a life on its own on if you kind of see this notification on Twitter, etc. People have given us a lot of character. But anyway it's we try to instill a lot of character and tone into our notifications and this is one of those that got a lot of attention. The other part I would say for us is it's not exactly a growth lever as you would think. As like something that that is a game mechanic is obviously building better courses. So our courses are never static. So if we have a French course, it's not the same course all the time. We keep making improvements to the course. We sometimes add harder content so the course becomes longer or we improve, for example, various sections of the course over time. If we can actually improve the content that makes it teach better and tailors the difficulty a little better to our learners, that also ends up helping our growth over time.

Got it. My guess is probably similar to like retention for a typical app where the first few courses really matter, right? Because my guess is a lot of people churning the first few courses. So make them as useful as you can, 100%.

I mean it's certainly well for us. Actually there's the when you're a brand new beginner, you know nothing about French and how we start teaching you matters a lot. We have another challenge which is not everybody comes with zero knowledge of a language. A lot of folks actually that are learning the language come with some level of knowledge. Like they might have taken two years of French in high school or their cousin was French, they learned a little bit of French. There's unlimited set of scenarios where you might know some. And I think one of the challenges we try to solve is can we place you to the right place in the course where you are, you're at the right place for how much. And I think that's the other thing that we try to solve, which is tailoring the experience to you based on how much French or if you're like intermediate, advanced, can we get you to the right place? It's a very hard problem actually, but it certainly helps retention if you can kind of land the right Person based on what they know to the right part of the courses.

Got it. And you've also done some pretty, like, deranged things. Like, I remember you changed the icon of the app. You changed the owl, right? You changed the owl to some, like, old Owl or like, someone like the icon. The icon changed after people didn't use it for a while. Is that one of the things that you did?

Yeah, we've done that multiple times now, actually. And we've changed the app icon to effectively a state where Duo looks messed up like that. That's the intention to be very true is Duo has. Has taken on this personality a little bit, where some of it is our creation, but some of it is our user's creation of this. We initially designed Duo to be a motivating coach within the app, but social media made it like, this haunting person that if you don't do your lessons, he will haunt you down. That wasn't necessarily our creation, but we certainly, over time leaned into it. And anyway, when Duo's face icon changes for the entire user base, so that's what the change is. We change it for generally, like, a span of two to three weeks to be messed up. Like, sometimes we made it, like, really wrinkly. Sometimes we made it, like, melty is what we called it. The face is melting. The last one we did is, like, we made his face look like he got sick. Like, he had, like, boogers coming out of his nose. The trigger this has created is for users who have maybe, like, don't have the daily habit, and they maybe forgot to do their lesson that day or that week or that month. They felt like it was personalized to them, and Duo is somehow pissed off at them for not doing their lesson. So they come back, think it's not personalized, but it feels personalized when you feel some amount of guilt that you didn't do your Spanish lesson and Duo's reacting to you, and that feeling generally creates. Well, people come back to the app and do their French. So that's kind of why we've done it.

Yeah, I think it's brilliant, man. Like, I think every other company is making their logo more and more boring. Right. And you're giving Duo, like, some personality. I mean, that's genius, I think.

Yeah. And one thing we. I would say we take pride in, which I've realized this is so normalized at Duolingo, but when I talk to other companies, we use the term unhinged as a very positive thing. Like, when someone says, oh, that change is unhinged, at Duolingo, that means you've done a good job. Like, you've really unhinged. This change, like our app icon is one we. When Duo does something really weird in our app, like, we're like, yeah, cool. That's really unhinged. And I was, when I talked to other companies, I say, yeah, like one of our product philosophies is making things unhinged. People are like, is that a good thing? Like, do you want to make the product unhinged? But I think that's in a way our philosophy. I think it. It really character when you're building product by kind of breaking these standards, kind of sanitized product norms.

Yeah. Like, I'm curious if this just sort of talk about this icon thing more like, does this come from bottom up or like, did the CEO want to do this? Because like a lot of these big companies, they're like, it goes through committee review. Right. And like something like this probably not make it through.

Honestly, I don't quite at this point with a lot of ideas that circle around the company. I don't know actually what. Where the origination was. But that being said, we have a very centralized product review process. And I can assure you, when we reviewed the app icon and the feedback that we gave. So we have a product review process where our CEO is involved in product review. I'm involved, our head of designers involved, and there's various product leaders that are reviewers. The feedback we gave was the there was not enough booger. It wasn't messed up enough. So it's like, there's certainly a lot of we're not trying to build just hygienic product. We certainly have embraced the kind of the unhinged product building philosophy in our review process as well. I think this idea itself originally was bottoms up, but then eventually we got really excited. So I don't know where it bounced in our process, but there's a lot of ideas similar to this where someone was like, wait, games, for example, do a lot of icon changes. What if Duolingo did that? And it probably baked over time into everyone's conscious.

Yes. It sounds like a fun place to work. I mean, power views.

Yeah. Unhinged, I would say so that we strike well.

Okay, so let's talk about how you guys grew revenue too, because it's still growing like 40% year over year. And I think the team tried many different things right before finding this freemium model that works. Maybe you can walk us through how this whole journey that you went on.

Yeah, Definitely. And this, this ties back to my personal story at Duolingo a little bit too. So I joined 2000 in, in 2016 when Duolingo had no revenue. So Duolingo had a growing user base. And the wisdom at the time for most consumer products was just grow, you'll figure out and monetize later. But 2016 was already five years into Duolingo's existence, so it was time to figure out how to monetize. And I worked on our three business lines, so I worked on ads, I worked on in app purchases and subscriptions. It's funny because if you go way back and find Duolingo's launch videos, a lot of these have now been taken offline. Our CEO Luis has talks where he's saying Duolingo will never have ads, never in app purchases and never subscriptions. And those are the three things I built in that order. And the reason that he had this, at least these statements at the time, was education. The common wisdom was you monetize your content. There was really a. And there's still not many products that can do freemium. But education is mainly monetized through. You lock your content behind the paywall and you charge for the content. And in education, people pay for the content. That's the standard exchange that everyone is used to. We never wanted to do that because we stand for access to education. So we always wanted to provide the education component of Duolingo free and wanted to figure out how to charge for the bells and whistles of Duolingo. And that was why we tried ads which you ask no one to pay for anything. And then app purchases, which were very optional things like repair your streak if you give us $2 type of purchases, then subscriptions and subscriptions when you have ads. The easiest feature to start with is no ads. And then we kept adding more and more features into the subscription that's, over time, defined our business model to be effectively freemium. And one of the benefits of that is we still don't charge for learning today. So you can learn if you pay or not. You can learn the same French either way. So our users who pay and users who don't pay learn the same amount of French. And it's been a lot of trial and error. Honestly, I think we. It's not, in hindsight we can say, yeah, we had like this exact vision. I mean, we kind of did, but we had to test what works, like how much ads hurt retention versus also piss our users off. What kind of ads work best? Does it work on Session start or session end. Like how much ad load is the right balance for an experience that we want to make really delightful. And related to that, how you talk about the subscription, if you call it plus versus super, what color should your purchase flow look, anyway, there's an unlimited amount of a B test we've ran to figure out what works. And that's been our journey, I would say to get to where we are now, which now subscriptions makes up more than 80% of Duolingo's revenue and it's grown quite a bit over time. So anyway, so I think it hasn't been like one change. We did it, nailed it. I think it was a constant experimentation that changes the feature set, how it's promoted, how it's talked about, who it's surfaced to, et cetera.

Like, I do think like the freemium subscription model is pretty common in gaming too, right? And in the future you can have how gaming subscriptions people complain when it's like pay to win. But a lot of games are doing like more aesthetics based subscriptions. So maybe you can like let people like dress up the hour or something like. Yeah, or maybe you already have features like that.

Yeah, we, we don't. I mean we actually used to have outfits for Duo, so that's a feature we used to have and now we have. For cosmetic features, we have avatars. So you users can set avatars on their profiles. These avatars would appear in our social surfaces. We have a set of social features like on leaderboards. This avatar is the one that represents you. There's a feed where it's very. We have a congratulatory feed. So if you complete 100 day streak, I can congratulate you and can see my avatar congratulating your avatar. Anyway. We certainly have, I would say, ambitions to grow our social surfaces and connect it to that. Potentially build more cosmetic items to. To make these kind of visuals more exciting.

Yeah, that makes sense. And like going to more recent years, there's been a focus on international expansion, right? And like a lot of companies when they think international expansion, they just think like, let's localize our content or something. But I think you guys have figured out like a more holistic strategy even beyond the product. Can you give a preview of that in a certain market?

Yeah, definitely. So I think we do a few things. So when we think about a market that Duolingo doesn't have as much awareness or adoption, we end up doing a few things. One of them is we want to make sure we offer the language Courses that market might need. For example, if you want to grow in Japan, if you don't have a Japanese to English course, meaning teaching English for people who know Japanese, you're probably not going to grow. So we want to make sure we have content coverage for us. So the product has to work for this market. That's one, two is generally solve any product issues that might exist for to to give some examples in China you end up having to do very custom things for your product to be able to work in China you need WeChat login, Google login doesn't work, etc. So does the product work properly in that market? And let's solve all the either the performance issues or things that are blocker to usage is step number two and I'll talk about all the product levers and I'll talk about marketing levers second and then the third is making sure the product is localized properly. Sometimes when entering a new market and if the market adoption is low, it tends to be the case that even the translations of the UI strings that we use aren't exactly great. So the localization of the product itself has to be good. But assuming the product works perfectly and you've tackled those, I think the second one is just getting adoption. I think we do a set of few things. We don't spend heavily on marketing in any market we are even for priority markets that we want to grow in. One of our marketing playbooks is always doing unhinged stuff on social media. TikTok has been a big lever for Duolingo. Well it started with the US but now over time we've also built local TikTok accounts that just build their own TikTok content. That's the similar unhingedness that we do in our main account from the US but they do it in Japanese in Japan etc. So a few key markets actually now there's there's the number of TikTok accounts that are localized are growing. So anyway social media and making sure it's local, custom and unhinged I guess is one part of the playbook some amount of performance marketing especially in the earlier adoption days we do but it's. It ends up being a small percentage of our growth but it certainly helps kick start more adoption. Depending on the market, the marketing playbook changes. For example, in Japan we do TV because TV is a is an important channel in Japan but we don't do TV anywhere else. So depending on the market and what might work there we do that and then in some other markets again Depending on what works in that market. I think that's the thing we try to learn first before we head and apply the same playbook everywhere. We also work with social media influencers from time to carry over our unhinged brand into their audience.

Yeah, you just want to become unhinged and go viral, make some memes and that's how the word spreads, right?

I think it's, it's. I mean, yes, in a way, I think that works. And it's not only we do unhinged stuff just to go viral. I mean, that's certainly part of the theme, but it really also builds character, meaning it also builds brand affinity. If we were, if we didn't have Duo, and if Duo didn't have so much character, duolingo wouldn't be a loved brand as much as it is now. So it builds this brand love for Duolingo itself. When we have, when we take a stance on our brand and character, rather than being super sanitized and say, yes, we are the education app, so we never want to take ourselves seriously because it really builds a better brand. If you're a bit more unhinged and have more character is at least our belief.

Yeah, I totally agree. Okay, so let's talk about AI. So I saw some random tweets from random people saying like, oh, OpenAI now has real time translation. Like, it's game over for Duolingo. I'd love to hear thoughts on that. Or it's like just people's concerns.

I love this question. And we get this a lot. And what OpenAI is able to do is also amazing. That being said, I do also believe when people say duolingo is doomed, they're making this assumption that no one will need to learn languages. I think that is a very unrealistic assumption and here's why. Real time translation tools have been around for a long time. Like Google Translate can do real time translation quite well actually, and it's been able to do that for a long time. That has not reduced the need to learn languages. The same the demand has for learning languages has only gone up over time. It certainly has not gone down. This is very similar to having calculators does not reduce the need to learn math. It's a very similar problem. And with real time translation, the one other challenge that you end up having is even if you can do the most instant, the best translation, a lot of languages, you need to complete a full sentence, receive that sentence to be translated back to a different language. Because the order of where the verb is and how it's structured might need to be reversed for the other language. So you end up having this lag. I'm saying also to say there's still. Even with the best translation services, there's still the challenge that you can't go get a knowledge job with a simultaneous translation service. Because would you want to work with someone that has a, I don't know, a 15 second lag in every interaction for now that I don't think humans want that today. That's one. The other one is the. The other challenge is there's many other reasons to learn a language other than just be transactional with other humans. It's very hard for us to even pinpoint like the top reasons to people learn a language on Duolingo because it's incredibly long tail and branch tail. They might connect with their heritage. They might have relatives that speak Spanish that they want to get a few words with. They want the authentic experience of ordering their cappuccino in Italian. All of these are still not solved with real time language translation. The second part. So we also get. Then I'll expand on your question. Is not just translation but oh, now AI tools can teach language. Like you can actually learn language by talking to ChatGPT and it's awesome that you can do that. I'll refer to my original example. There are also books that teach you a language. Why have books not disrupted language learning? Because it's boring. So unless every experience we build on top of our learning, which is the every kind of fun element we talked about, even including duo doing really weird shit to our streak, our game mechanic and the whole experience of Duolingo is really one of our core differentiators. Like we make learning fun. And until a tool can do that very well, it's hard to see how Duolingo would get entirely replaced with AI.

Yeah, I think learning a language or learning something new is like it's like worth working out. Right. You gotta do it for like a long time to actually see results. So these other products are already optimized for that. So.

Exactly. And people learn languages like a lot of folks learn languages for brain training reasons. AI isn't exactly the simultaneous translation that AI can do doesn't replace that.

But AI is probably a game changer inside Duolingo itself. Right. You're probably using it to do a lot of stuff. Maybe you can cover some how you're using AI to make the product better.

We do use AI for sure. I would say we use. Generally AI is used for two main Reasons at Duolingo today, the list is technically longer. But the two big ones that have been, I would say, transformational, at least in the past few years is one is content creation and the second one is building interactive features using generative AI. Content creation. If you think about a duolingo course, so let's say the duolingo course that teaches English to French, it used to be the case that it would take many human hours to write all that content out, and then if you also want to change it, you would rewrite the whole thing out again. Like if you want to ab test, for example, a new way to teach past tense, you would rewrite the whole content again. So that's the. It's many hours spent and any improvement you improvement you make is again, many hours spent by humans. Now we can use. We're not at a point where you press a button and you generate a duolingo course entirely on, on generative AI. However, a lot of our hours have now been reduced because you can generate much quicker content using generative AI and then you can prompt and edit a bit to get to final state rather than doing everything by hand. So content creation has gotten much faster and it has enabled us to launch courses faster, make improvements to the courses faster. So the whole machine of content creation at Duolingo has gained insane speed in the past two years because we now use generative AI to generate a lot of our content. The second one is, I would say, interactive features. So one feature we're very excited about that we launched recently is what we're calling video call. So it, it mimics exactly a video call experience where you call one of our characters, her name is Lily. She is, the character's vibe is she's an emo teenage character that kind of doesn't care is her vibe. But that's the kind of the fun character we gave Lily. But the idea is you call Lily, since we know how much Spanish, because we know how where you are on your course, Spanish or French, whatever language you're learning. We tailor the vocabulary that Lily is using and the speed that she talks to your proficiency level. And you can practice conversation with her exactly as you would with a real human that you would have video called in Spanish or in French, whatever language you're learning. It's really been transformational to teach speaking. One of our challenges for teaching conversation has always been mimicking a real life conversation. And previous kind of chatbot like experiences we've created were always kind of too dumb and. But I think now with the GPT4s of the world. It's actually quite smart. So we can tailor this and you can have an almost an entirely freeform conversation that's tailored to your Spanish level. So it's not too hard, it's not too easy using AI and that's unlocked this massive use case that we always wanted to get better at, which is conversation practice.

Yeah, yeah.

So anyway, long story short, those have been quite transformational to how we build product here.

Yeah, I'm like really impressed by it. I haven't played with it myself, but I saw some videos.

Oh, you have to. It's really good.

It's good. And yeah, it's like it has like lady seems to have a lot of really good personality. Maybe you can give us like a brief inside look at how this product was built. Like what have been some of the challenges? Like any kind of story you can tell around this?

Yeah, yeah. So the whole story about this feature actually started with another feature we have which is called Roleplay and that is a text based version of what you. It's not exactly video call, but it's a text based back and forth where you practice conversation. And that was. Now we had, we rolled that. It's called Role Play. It's part of Duolingo Max, our subscription product. And it's. We put you into a scenario and say, okay, you're at a cafe, order coffee from this barista and then you chat, you effectively type and you go back and forth ordering your coffee. We realized that was really good for producing language. But since it's typing based and we gave, we still give you the options to speak and type, but when you have the typing option, you people are just gonna type. Because imagine a language you're not comfortable in. Typing is going to be much easier than just speaking out loud. So you people default to typing. And typing does not train your conversation skills as well. So when we built the typing version, we really liked it and it's still live, but we thought we could improve it and over time as also the LLMs and generative AI improved the output they provide the latency of these products, we wanted to prototype something that is speaking based, like full speaking based. And the one thing we wanted to avoid is making it look like a Duolingo lesson. We really wanted to mimic as real life as possible. So we prototype ended up prototyping a much uglier version of what we have now, which was you talk to Lily. There's a phone call sound. So when you call her feels like a call she she answers the phone, and then you're in a conversation. The. The thesis is we want to mimic real world as much as possible, because that's when you really learn how to converse. If we make it easier and we give you a lot of scaffolding, for example, we don't put text. We. You cannot read what she's saying. That's very intentional, even though it would make it much easier. It also doesn't help train your conversation skills as much on top of that, to make it also feel like you're calling your friend. She's actually a little sarcastic as well as you talk to her. So sometimes she will make sarcastic comments and spontaneously say teenage things, because that's our personality. Anyway. All of these have been intentional, and they've kind of been added over time. Like, we started with some amount of prompting to what we think Lilly should be, but then we've iterated on it. So we've been working on it for a year now. We've iterated on it many times, and our process has been, we have a thesis, we prompt it, we test it internally. If we like it, that goes out to our learners, and then if we don't, or we keep actually, like, our internal builds almost every few days, just get a new version of Lilly that we're testing. We're trying to make it more delightful, smarter. One of the new additions we did was memory. So she now has a memory of your previous calls. So if you tell her, I'm going to Mexico next week, if you talk to her next week, she'll be like, are you ready for your trip to Mexico? Are you in Mexico? Anyway, so she kind of keeps track of what you've told her, which has been a big upgrade. Anyway, the list goes on. But I think our goal is to continuously make it better because we really believe this is the future of how people will learn how to speak languages.

Yeah. And did you also, like, roll out some beta partners or beta customers and. Or like, how do you know when it was ready for release? Because this AI stuff is, like, very iterative. It can go off the rails, or maybe you wanted to go off the rails, probably be unhinged. Like, how do you know when it's actually ready?

Yeah, we want. We certainly are okay with unhinged, but we certainly also put some guardrails so it doesn't hallucinate. That being said, so we work with OpenAI for this feature video call. And I mean, they. I think they also care a lot about guardrail, so they. They put a lot of Guardrails to, to the output. That being said, how we test, we first test internally, everything. I think that we now have 800 employees. Not everyone tests every feature. But a lot of our employees are very much daily users of Duolingo. We have, we keep streaks as well. In internally we carry a streak is a bit of a social status here and inside the company. So a lot of people test our products and dog food. So dog fooding is phase one generally, if dog fooding goes well, and that's where we refine a lot of our features, we do AB testing at small rollouts. So we do a small rollout, see how it performs both qualitatively and quantitatively and then increase rollout over time. So I think that's been our approach.

Okay, let's talk about your product culture a bit more. Just a quick wrap up. Like I've worked at companies where like, like PMs have this like love hate ratio of AB testing, right? Because like you can easily just a B test something shady and then you just launch it to get some quarterly milestone or metric, right? But so you guys have this really strong a B testing culture, but you also believe in taking a long view. The app is beautiful and fun. So how do you kind of balance these two things? I think a lot of companies have trouble with this.

Yeah, I think. Let me see. I don't know if we have a silver bullet, but you are certainly right that we try to balance rapid a B testing, but not rapid a B testing our way into insanity and shitty product features. So one is cultural. So we really have instilled from our CEO to leadership and to every product manager, designer, engineer, so everyone that works on the product, that quality matters. So it's, we wouldn't roll out something or we wouldn't roll out or launch something just because it moves metrics. Because that's how you test your way into really shitty product, right? Like you can send an extra notification that will move your daily active users. You can send one more, two more, five more, ten more. But at some point people are going to delete your app. So there's certainly going to be. AB testing will drive you to insane, weird, shitty experiences. So I think what we do generally is culturally we have this understanding that like we got to hold the bar somewhere related to culture. We have a central product review process where the same group of leaders review every change that go into the app. So that builds consistency. So generally. And product review holds a very high bar for introducing something that might feel spammy to our users. So adding an extra notification, another badge somewhere, et cetera, all of these, we take really great care and end up like if, even if teams want to push forward, we end up saying this is too spammy, we should not do this. And even we don't end up saying that a lot. I think a lot of the teams have instilled that they like, we really care about building quality experiences. Yeah, that's the other one. Three is, I think it's. We have this operating principle, this is maybe goes back to cultural and connects that we call take the long view. So you can always optimize metrics in the short term. Like you can get more users by sending more notifications, you can increase your revenue by building kind of like slightly obscured purchase flows, etc. But this really only helps you in the short term. But everybody knows that if you want to build a 100 year company, which is something we have, we aspire to do, you have to build things that are tasteful for the user. And I think that's ingrained in, in I would say kind of everyone's mind.

Yeah, I think like Roblox, the company I work for, also has the same view. Take the long view. The exact same principle, actually.

Is it the same words? You guys say, take the long view.

Take the long view.

That's awesome.

Yeah, but, but I think it's, it also comes down to incentives, right? Like, like, I think like, you know, do you promote these PMs for like not shipping anything? Like I think there's a really good example someone gave me at Roblox. Like at Roblox you can like not ship anything for several quarters or not move a metric for several quarters and then in the fourth quarter or like then all of a sudden it moves a lot. Like that is something that you don't have to like try to optimize, try to improve everything every quarter. Oh yeah. I'm just curious, like now that you're the head of the product, like, how do you reward the right behaviors? It's just kind of common sense. Do right behaviors.

Yeah, no, I mean we reward the combo of both business impact and quality product building. I think in the absence of one and just the other. For example, if you work on a lot of stuff and you build really quality things, but you don't ship anything and you don't execute well, that's not a, that's. We can't call that a win. Or if you have a lot of business impact, but you really tried spamming your users to get impact. Well, we can't reward that either. I think we want to reward the combo of you will understand what a quality experience for users mean and using a high quality bar, you build impactful features that drive business impact. That's what we really reward for product management here. And I would say it's worked out well. And PM design engineering, very similar philosophy. So our product teams end up having to balance both. It's not super easy, so I'm not going to make it sound like we figured it out. It's the perfect formula. Sometimes it does clash when you have a business metric that you really want to drive, but I think generally that tension between the two or tension or kind of the balance works out well.

And like reflecting back on your eight years at Duolingo, I'm sure you've managed a lot of PMs, coached a lot of PMs. Do you have any like closing words of advice for product folks watching this interview? How they can have a fulfilling career, how they can become a little bit unhinged? Do you have any good advice for them?

Yeah, a few bullet points. I always end up saying one of them is, especially if you're building consumer experiences, understanding mobile design, if you're building mobile or just general good user design becomes a major differentiator for product managers that go a long way. I think there's a lot of product managers that I that I think end up having a hard time when they are so removed from the pixels. I think good products really live and die in the pixels And I think PMs can't just be at the spec level. Like I wrote the strategy, I don't know what the design is going to look like. Can't be a winning combo for PMs. That's certainly not the case at Duolingo. So having a good eye or a good understanding of good user experiences really helps. Certainly we don't expect our PMs to be like designers themselves where they can go design the best flow, but they should really understand what great design looks like and they can tell like if the copy off is a little bit or if a pixel doesn't look great, they should have that eye as much as or as close to as a great UX designer would. So I think that's one advice I give and that's one area we train a lot of our PMs to get better at as well. That's one. The other one is optimizing for business impact. So I think we talked about this a minute ago, which is a PM career arc. If you look at PMs that have done various things in their career. A lot of times you want to make sure you can. I mean the one metric that really matters is driving business impact and making change. Business impact doesn't have to be dollars, doesn't have to be users. It could also be solving a hard business problem. But I think you want to be at a place or at a team, at an org, at a company where you feel like you're working on impactful things. I think that really matters in building a good career arc that you feel good about. And that will teach you more than like for example, projects that take a long time and might not be the kind of the impactful work that you're excited about. So being excited and having kind of clear impact on the company or the product is matters and important. The third one is I always recommend Optimize for learning. And the one of the best ways to learn is trial and error. So product management, you can't read a book or take a course and learn. How everybody learns is apprenticeship. And apprenticeship is getting your hands dirty. And to learn faster. What at least we have seen is the more trial and error and at bats you get, the faster you learn. So the more experiments you run, the more specs you write, the more product reviews you present. Anything, like anything that comes with the core job. If you can do that at a higher volume, be there. Like do it at a higher volume because for sure your career will move faster.

Yeah, I love those. So like just kind of recap like you got to have a high design taste, high UX quality. You got to have impact. I mean you got to ship. You gotta ship.

Having packed, you gotta ship. If you don't ship. Exactly, yeah, yeah.

Cause I think that's really good advice because a lot of times like it really runs me the wrong way when like a lot of times these PMs just like they have some like strategy document or like some like three year plan or something and they just like spend time like, like they don't give a shit about the actual product man. They don't care about the copy or anything. Just get around the strategy document and I'm like dude, like next month you're gonna learn something new that completely invalidates your three year strategy. So.

Exactly. The strategy documents go kind of useless in three months. Has been my experience, especially if you're at a fast moving org or. The other thing that rose me the wrong way is I see a lot of PM resumes that say their main accomplishment was aligning stakeholders like presented to this documents go kind of useless in three months has been my experience especially if you're at a fast moving org or the other thing that rose me the wrong way is I see a lot of PM resumes that say their main accomplishment was aligning stakeholders like presented to the CEO what was the product you shipped so you're the career arc that really matters for product management is what have you shipped.

Yeah man. Let's change the culture together. I'm glad to have this. Yeah cool. Well this is awesome. Where can people find you Jem Jem or learn more about doing I am.

On LinkedIn and Twitter both active on both so people can find me in either of those places. It's Jen my name and last name so Jem Khonsu in either of these platforms. Awesome.

This is awesome chat. Thanks so much for your time Tim.

Yeah thank you Peter. This was awesome.