Gina Gotthilf (ex VP of Marketing/Growth at Duolingo)

Hey everyone, this is Erik Torenberg and you're listening to Execs by On Deck. Execs is a show for founders, operators, and leaders who want to understand the playbooks, frameworks, and tactics that leading tech companies today have used to scale. Our guests are tech execs in key roles at top tech companies who share their hard-won, earned secrets on how to scale faster. Gina Gotthilf spent 5 years at Duolingo and helped the company grow from 3 million to 300 million users. She's the kind of VP of marketing that isn't afraid to get her hands dirty. Gina is an A/B testing vet and shared her best frameworks and lessons for how to A/B test successfully. She also shares her PR and influencer strategy tips, as well as her thoughts on how to hire your first marketing person. She's now co-founder of Latitude, which helps early-stage entrepreneurs in Latin America solve some of the biggest problems in emerging markets across the globe. Her passion for her job and life are evident throughout this episode. Hope you enjoy. So Gina, you have extensive marketing and growth experience, and now you're also investing and advising startups. I'm curious if you could outline a little bit for founders who are maybe seed, maybe they're just getting to Series A, and they're thinking about their first marketing hire or their first growth hire. When is the right time to make that hire, and what should they look for in that hire, and how to set them up for success?

Yeah, that's a great question, and there's not a very clear answer because of course it depends. But what I would say is So I think first of all, it depends on the nature of your product and who you have on your team. So if you're going D2C, if it's a very consumer-heavy product, you probably want to be thinking about marketing and brand, honestly, I think from day one, unless you have a product that is in itself completely groundbreaking, like paper, like you've just invented paper. An element of brand and marketing is a really big part of what the product is and what it's going to become. That said, I don't think that everyone needs a marketer from day one, obviously, and that's a luxury. I would say that for early-stage startups in general, to start out with marketers who are generalists and who have experience working at really tiny organizations or scaling really tiny organizations, because working as a head of something in marketing at Google is very different from leading marketing at a tiny organization that doesn't really have a brand yet. You know, when we hired at Duolingo for PR, for example, our first hire, we were really reticent about hiring from Google because we knew that PR people at Google mainly spend their time trying to prevent sources from writing wrong things about Google and not like chasing reporters to beg them to please write about Google. It's a very different job, and that applies to marketing as well. So I think that's step number one, someone who's really ready to get their hands dirty and who's ready to test a lot of different hypotheses in a really organized way. And once they're able to find out which ones of these hypotheses work out or basically which channels work best for that startup, then you can start thinking about more specialized hires and people who really understand that a lot more. So there's no right time. Hire, I think, someone who understands a little bit about marketing and brand as early as you can, especially if you're not inventing paper, and go for a generalist would probably be my advice.

Why don't you unpack the story of what that looked like at Duolingo? In terms of what were the early things you guys did and how did that evolve over time?

So I think I did some cool stuff at Duolingo and I take some of the credit, but really Duolingo is an incredible organization because there are a lot of amazing people there. And one of the co-founders, Luis von Ahn, always had, I would say, an eye for a lot of things that were outside of his scope as someone with a PhD in computer science. So, he definitely thought about brand from the early days and that infused how we thought about things as well. In my case, I was the second marketing hire, actually, and here's a mistake that Duolingo made early on. They hired a great VP of marketing as their first marketing hire. I think they would just raise a Series A and this professional was amazing in so many ways, but had not come from a startup background. And so, Their approach to prioritizing what needed to be done and to trying to delegate rather than just jump in and do actually ended up slowing down the organization, which is my luck because this person hired me as their little Brazil lackey. So, I was a Brazil consultant to try to figure out how to help Duolingo grow in Brazil. And the company realized after a few months that they were only growing in Brazil. So, you know, it was my luck, but it was also because I was, I don't know, 27. Super eager to hustle and do whatever it took. And so, that's kind of how I started my career at Duolingo. And early on, I was trying to do the same that I had done previously with Tumblr, which was, how do we get attention for this product? How do we get the maximum number of people to discover it without using any marketing budget? Because I had no marketing budget, both at Tumblr and at Duolingo. And so, then the question became like, okay, how do I hit mass media? How do I get Duolingo on mass media, meaning like big newspapers, big magazines, big websites, and not try to like build audiences, but just leverage existing ones. So what would be a story that a journalist would want to write about and then reverse engineer that? That's one thing. The second thing I think that, that I focus on really early on, and again, I credit Luis with a lot of this, is thinking about the brand and what it means in terms of the mission. So I think a lot of companies and Erik, I think, you know, you, you think about this a lot and I really like how you Like the things you've said about thinking about the mission. And even when you talk about On Deck, a lot of companies think that the mission is kind of like this nice-to-have thing that motivates employees and like sounds really nice on marketing. But really, you need to be looking at KPIs and you need to be driving numbers and all of that. And it's not that you shouldn't be looking at KPIs and driving numbers, but the mission and the real mission becomes the brand. It becomes who you are. It becomes who your customers are because they believe not just in the product that you built, but in the mission behind the product, I would say 50/50. And so understanding how to talk about your company or your product or whatever it is that you built from, from that perspective and being super consistent with your messaging can, can have outsized results across channels, be it PR or be it social media or your own content that you're producing. So I think that that would have probably been number 2. So Duolingo, free language education for the world. That was always— everything we talked about was the mission to bring free language education for the world. Because people didn't want to promote a new language learning app to their friends, but they did want to get behind a mission of changing the world and changing education. And so, that allowed us to get our message out there. It also means taking risks. So, for example, when Trump decided to close the borders and there was all this anti-immigration stuff going on, we decided to go full-on, like, Duolingo is for and by immigrants and we're all about diversity and, you know, we're built by immigrants and like doing social media posts around that, trying to keep it not political and just be like a celebration of like diversity, but at the same time taking a really strong stance. And so understanding who you are as a brand, I think is really important as well and was part of the original work, original work we did. Although it wasn't like we sat there and we were like, who, if we were a person, what would we be like? Like we didn't do that exercise until like 2 or 3 years later. But we developed a really strong sense of self and maintained a very cohesive messaging from the very early days.

Thanks for all that overview. You said that you did a lot with media. If this was 2022 and you were in the same position that you were back then, knowing how the internet has evolved, what might you have done differently? Or would you have done the same strategy?

You're making me actually think in this interview, Erik. I was hoping I could just say things that I'm used to saying. But I should have expected that. You know, look, I don't know, but I would go with the same way of thinking, which is I'm going to go and try to hit the largest number of people I possibly can most effectively. And so where are those people? And I think, again, like a lot of startups are like, I'm going to start building my social media. And it's like, yes, you have to start building your social media, but that's a very long-term strategy. Where are people today? I think obviously today, you know, social media has evolved a lot and then you have all influencers, not just like the professional influencers, but like everyone is an influencer kind of way. And so figuring out how to leverage— I would probably come up with a number of criteria, I think, and I would try to find channels, meaning either people like influencers or channels or media that fit those buckets, meaning like maximum number of potential audience and then maximum fit in terms of what they care about. And then ease would be How easy do I think it'll be to get this person to actually feature us in a way that we would like to be featured without having to pay anything? And so that depends on who they are and what their motivations are. Yeah.

Okay, so you're the second hire. You're, in your words, the Brazil lackey. You're figuring out growth in Brazil. They realize that they're only growing in Brazil. Then what happens? Tell the story of what happens next.

Cool. That's an easy one. Thank you, Erik, for that easy ball. I can just talk about the past. Yeah, so, so, so they hired me and I was doing all this very sort of hands-on stuff trying to get Duolingo featured in any newspaper, magazine, etc., across the country. Worked out. And then, and then I did some meetups also. So we started cultivating the community, but that was honestly kind of not the priority. We were doing that kind of as a nice-to-have. I think, you know, you and I can talk about this for hours, but community back then was really not important in the way that it is today. So we started growing in Brazil and the strategy was working. So actually we doubled down on it. We were like, let's do more of this. seems to be working. So, we kept— which basically meant finding people and places that would understand why Luis von Ahn, the founder of Duolingo, was amazing. And it's not that it's hard to understand, but most people didn't know what Duolingo was. Most people had never heard of him. And so, figuring out who would really care. And so, for example, he's an academic. So, if I go to the top university in a country and I find their most incredible computer science professor and I tell them about Luis and I say, hey, do you want Luis to speak to your students? Or something like that, it's a no-brainer. Now we have someone inside the organization that's gonna fight for Luis as a speaker and talk about his prominence. So we have an internal, I don't know what the word is, like mole kind of thing. So that was one approach. And then also finding shortcuts to make journalists understand the importance of Duolingo and Luis Fuanon by finding names that they attributed a lot of importance to. So top university is one of them, top tech conference, anything that sounded like cool and big and important, we would use that. So it's not the same as saying like, hey, Brazilian journalist, Erik Torenberg is in Brazil, do you wanna interview him? But if I say, hey, so Erik Torenberg just spoke, gave the commencement speech at Harvard, Would you be interested in talking to him? That's a different— I don't know who Erik Torenberg is. I've never heard of Andech maybe, but I've heard of Harvard. And if Harvard picked Erik, then I need to pick Erik. And so creating that as much as we could. So it worked really well. And so they asked me to do the same thing in Chile, Argentina, and Mexico. And me being this eager 27-year-old, I was like, yeah, sure, I got this. Luckily, I had done a little bit of that at Tumblr, so I had some connections and kind of went off of that. That worked out, so they're like, okay, so, so, you know, let's double down on Mexico, and now we need you to go to Europe and Asia. And I was like, yeah, yeah, of course. And so that was pretty crazy, I think a leap of faith, and they asked me to become their head of marketing. So, I was traveling around Japan, China, Korea, India, Russia, Turkey, France, all of these crazy countries, and to figure out how to make Duolingo prominent, how to get the maximum number of people to discover and use Duolingo. And I had the benefit of working with a product that was awesome. Even back in the day, like, it was a product that if people discovered, there was a very high chance that they would come back tomorrow. And so, that's the only reason why I could do my job in the first place. So, we did that. And then, so then the whole reverse engineering PR thing was interesting because it led me to a bunch of other strategies. Like, you know what the newspaper might want to write about? The government. So, how do we get Duolingo involved with the government? You know, like, okay, well, if the government partners with Duolingo and brings the platform into their public schools to teach English more effectively, now this is a story. What else is a story? And so then I'm working with partnerships and, like, government relations and, like, all this other crazy stuff that I didn't really have a name for, but it was just always thinking about the endgame, which was getting to the maximum audience size. But even, like, getting, I don't know, someone really impressive to talk about Duolingo or Luis or to have them over, If there wasn't a massive audience tied to that strategy, it became just a badge of honor and it didn't work. So it was always like, what's the second step? And then I continued to manage sort of brand, social media. Then I took on newsletters and our email marketing. And then they asked me to become their head of growth. And that's a funny story because I remember when they came to me and said, hey, Gina, we were thinking, actually, our investor, Leila Sturdy at Google Capital, I think, was like, Luis, you're the person who keeps looking at the DAUs obsessively every day. you need someone in your company to do that for you so it's not just you. So they like considered hiring someone and then they decided, okay, Gina seems like things, take things and run with them and we're gonna, we're just gonna try this. And I was like, yeah, yeah, that sounds amazing. And then I went back to my desk and Googled what is growth marketing? Like had no idea. It's also funny 'cause I'm sitting here with, and I'm gonna do a little promotion. I'm sitting here with Andrew Chen's book.

Yes.

Because I discovered Andrew Chen back then and Andrew Chen's blog and I'm like, okay, growth. And then they actually like, you know, I think Layla introduced me to Andrew because I was living in San Francisco, opening Duolingo's office there, which never happened, but then trying to hire engineers and figuring out if we needed a San Francisco presence, but really understanding what growth was. And then they were like, okay, Gina, here's an engineer. You're the growth team. And I had never worked with product before. I'd never worked with engineers, designers, product managers. And in marketing, I think it had, at least until then in my career, been very common for me to be on the outskirts of product. We made this, go sell it. You know, and so, and honestly, we, as I think a marketer, I always felt very devalued from that point of view. So I was like, I get to work with product. That's amazing. And then I was like, I am so scared. I have no idea how to talk to this human being who was like, Sven, I hope he hears this one day. But he was like the ultimate German engineer of all time. So he literally didn't say anything that wasn't necessary. And like, So our interactions, and you know how I am, I'm very energetic. So I'd be like, hey, Sven, how was your weekend? And he'd be like, good. And I'd be like, what? So what did you do? Nothing. So like that, that's kind of how we started our relationship and started figuring out like, okay, well, what can we do? What can we do that requires just Sven? Because like Gina can't do anything. Gina can come up with ideas and help like manage and like negotiate and get the resources we need. But like, I can't do anything. You can code. We have no designer. All right, so what are we going to do? And so we started out with very simple things like adding a little badge to the app, you know, like the little red dot that makes people want to open it obsessively, like to get rid of the red dot, or like testing copy for notifications because I could write copy. And that started working out. So they're like, okay, well, here's a designer. What can you do? Because we'd be like, okay, we have all these other ideas, but we can't do them because, because we don't have a designer. Okay, so here's the designer. So, we started coming up with more ideas, things that we could change in the app and then the website, and then of course the email flow and all of that. And then from then on, the team grew. And so, I continued to be in charge of PR, brand, social media communications, and then also the growth team, and then got to hire people to, you know, more and more so that I could scale what I was working on. But until about year 4 of Duolingo, PR was just like, Me and a huge Google spreadsheet of names across the world of people I constantly pinged and tried to coordinate during launches and got really mad at when things didn't work out.

That's a great overview. So many questions from there. One is, how did you evolve from an IC to a manager?

I was an IC for a very long time, I would say. My first hire was an intern. He's now, I think he's still at Duolingo and he's doing incredibly well as a senior product manager. Zan Gilani. He was my first hire, and it was like just like fresh out of college. I don't know, he studied like political sciences or English or one of those useless majors. Like, I studied philosophy, you know, useful, useless.

Yes, yes.

So it was just like me and him, and like, I think that's when I really kind of started learning how to figure that out. My team grew very slowly because we decided to grow Duolingo extremely carefully because adding people to a team can add all kinds of friction and people have arguments and they don't want to give up their Legos and so on and so forth. So, Duolingo actually grew a lot more slowly than you would expect. I really became, I would say, more of a manager as I started overseeing PR. So, we hired 2 people to take care of PR and then grow out the growth team. And then not only did I become a manager, but I became a cross-functional manager, which to me was completely new. And so managing people who were way smarter than me and who knew so much more than me about anything that we had to do. So we had engineers who were like, you know, PhDs in machine learning with like perfect grades from Carnegie Mellon reporting to me. We had like these incredible designers, data scientists. And so that was, that took a lot of, you know, pardon my French, but like balls in a way and a lot of getting accustomed to because I was so intimidated originally and I was so scared that they would think I didn't know enough to manage them and that I was there just because like I happened to be there because I was, you know, outgoing stuff. So I quickly learned that my value wasn't to have the answers, but like in philosophy, to ask the right questions and to help people get along with each other. That sounds so basic, but—

Yeah.

When you're an individual contributor and you're really going after the ball and you're just like, I'm gonna go do that, I'm supposed to do that, it's really hard to look around and understand what others are doing and how to play most efficiently with others around you. And so my job became a lot of one-on-ones to really uncover what everyone thought, because a lot of people don't want to share in a group setting, for example, like in a meeting, or were having conflict with someone else. Or like most obvious example for me is situations where I sat down with the engineer and found out that they were going to take a whole like 5 extra days to implement something because a designer wanted there to be an animation. That kind of thing. And the engineer is sitting there being like, I must do what the designer said to do. And this is an animation and I'm gonna do this. And I'm like, we don't need an animation. We're just trying to test something. Like, take away all the frills and then going back to the designer and then being like, no, no, no, no, but I really want the animation. It's like, yeah, yeah, yeah, but it's gonna take 5 days. You know, and like making everyone understand sort of what the limitations are and helping that communication happen before it's too late. And then in communicating across teams, And I think becoming an advocate for the people who work with me, because I think it's so important to hire people who you think are incredible. And then you can say, get out of their way, but it's like, it's more than that. It's get out of their way and create opportunities for them to be at their best. And you can, because you can see further and you have access to senior management and like the vision and whatnot. So you get— you're instrumental in doing that. And if you don't, then you're stifling your own team. And moving a lot more slowly.

What did the marketing team look like at scale and how did you think about how big to make it versus how small to make it versus when to hire a specialist? Or how did you think about just team structure?

Erik, I think the team probably grew a lot more after I left. We were always going for minimal, minimal, minimal at all times when I was there. And in terms of hiring, we went about it like as we didn't think like we think it would be cool to have a person doing this, this, this. We would wait until it was unbearable because we were already doing it or we were trying so hard to do it and failing because we couldn't to then hire someone for that position. And at Duolingo, how it worked in many cases was the engineering organization was consistently hiring engineers. The design organization was consistently hiring designers. And so then they would get allocated to different teams. And so then you'd be like, I really need a resource. And they'd be like, okay, well, how about we give you half? I had to do a lot of halving. Which I hated as a manager. But you know, you get whatever you can take, which is like, okay, this this designer is going to do half growth, half like something else, and then we're like mad, you know, like begging for their time. But you know, then I think I think the more you're able to do with little, and the more results you're able to prove, and then the more you can you can show listen, I could be growing this much, or I could be going this far if only you I you got me this. Then it becomes a very easy negotiation. with the executive team. So yeah, we always erred on the side of super lean. I don't think that that's necessarily the right answer for everybody, but I am very scared personally or reticent as a manager of hiring too many people at the same time because I've seen so many people just like step on each other's toes.

Yeah.

Not on purpose. And you don't account for the whole human aspect of organizations. You look at a chart and you say like, here are the positions we need to fill. You don't think like the person doing social media is going to fight with the marketing manager because they both think that they own Facebook comments, you know, like, and this person is going to feel like they weren't selected for this position and that you hired from outside and they're going to get really upset. And like, it might be silly and maybe there's a way to overcome this, or maybe we're all going to be led by robots and this is not going to be a factor, but we're not there yet.

Was, was marketing and growth the same organization at Duolingo?

Yes, it was. So it was a, it was like a cross-functional thing, which got a little bit of a mishmash because we had like the PR social media people there and then we had like the product growth people there. And we did. So what I started doing was like 1-hour meetings, half hour we would talk about this and half hour, like whoever didn't want to be there anymore could leave and we would talk about this because I did think that it was important to keep cohesion and our designer was the same resource across. So like When I was prioritizing growth tactics, I had to throw in PR and social media initiatives into the same mix because we had the same people, you know, and I didn't, I could have just said like, hey, like we need 2 separate teams, but I liked the discipline of having to think about like, why are we going to go do and, you know, run the social media campaign or why are we going to go do this whatever PR thing? How many users do we think we can get? And that's bottom line. That was our metric. And so If the PR team thinks that they can get this many users because they can get the New York Times to write a piece about this and they estimate that there's going to be a 3% conversion from that piece and that of those people, X% are going to become DAUs and their number is significantly larger than the growth team that thinks that they're going to implement whatever and it's going to bring this many users, or if they can argue to me that what they're doing takes way less resources and time, in terms of time from our team, For example, badges is something that I often get asked about with Duolingo. That took a long time. And so, in terms of ROI, the I was super high. And so, when you have all of those different initiatives on the same plane, you're forced to think about R and I for each and every one of them. So, I personally liked that. And then I had the benefit of having engineers, designers, data analysts, etc., that came from other organizations working in the cross-functional team. But then I was the organizational manager also for everyone who was doing any brand marketing, copy, whatever related stuff.

Were you also in charge of determining which metrics to optimize for, or was that more the CEO?

Honestly, the CEO, I would say. I think, you know, there was never a moment where I was like, this is the wrong metric, we must choose another metric. DAUs was the right metric for Duolingo, and I would argue probably still is for a number of reasons. I was not the one who identified that. Luis identified that, and we were looking at the views from day one. And it came from a little bit of an obsessive impulse to consistently check metrics every single day and to see how things were going on a daily basis, which I think was super beneficial to Duolingo. But also just from the fact that, like, so messing with daily metrics is a lot like— fudging daily metrics is a lot harder than weekly, monthly, or like longer-term stuff because you can't— It's a lot harder. harder to be like, well, this week this, this week that, or like, let's, you know, like there was a holiday or whatever. Like every single day you're looking and you become really great at tracking trends in that manner. It also is super beneficial. And that's my main, my favorite thing about DAUs, that it allows for much more quick testing and much more quick iteration because you get results within 24 hours. And then of course you need to let it run for like a week or 2 depending on statistical significance and buckets and all of that. But It allows you to have at least one sample of results within 24 hours and then another 24 hours. If you're doing weekly or monthly, you need to wait a week or a month for your first set of results. And that really, really gets in the way of moving quickly. So, and then in terms of benefit to our users, so like I think that's a key one with regards to picking the metric. You didn't really ask me, but maybe you were trying to ask me that. is you have to think about what makes sense for your user, obviously. So not everyone's going to have a daily hourly user or a daily active user, right? If you're a bank, do you really want someone to be checking their bank account every day? You could, but you would have to make them really obsessed with their bank account. And that means that you're not giving them the safety, which you were probably there to provide in the first place. And so you need to think about what makes sense in terms of your mission, what you're trying to accomplish, and what your user needs and wants from you long-term in order to determine that metric. And when it comes to language learning, it's very hard to learn a language unless you practice very regularly. And a lot of people take language lessons like once a week or maybe twice a week. And it's just not the kind of thing that you can sit down, study for 12 hours, and then pick it up again in a week. You'll forget. So the daily practice is super important. It also has the benefit that if you have a daily practice, then that can become a habit. So we really can like work from creating a habit type perspective, which from the user's perspective, Helps because they're trying to achieve the goal of learning a language and this will help them. And then from the growth perspective, that's an amazing lever to play with. So yeah, there were a number of reasons why that made sense.

When you advise startups and you find yourself saying, hey, this is the wrong metric, where do people make mistakes when trying to pick the right thing to optimize for?

I would say 2 things. I guess many things. One is picking a metric based on either what you want So, of course, if I'm a founder, I want someone to use my thing every day. I want that. And you can come up with a million reasons why it's good for the user to use your thing every day, but you need to think about whether that's really the value that you're adding and if it makes sense. And then the second one is picking a metric based on, for example, an interview with me where I said DAUs and then picked DAUs because Duolingo did DAUs, right? But you have to think it through. The other one is, I think I see this a lot more and I think it comes from a perspective of wanting to be safe and to error less, but also of less pressure from the team that's picking the metrics. So not the founder, but the actual team, which is picking longer-term metrics. Like, well, we can't really expect our users to be back every day, so we're going to pick whatever, a monthly active user, right? And then like, yeah, okay, monthly active is good, but then you have to wait a whole month. To see how you're doing. And that's for a startup, a month is a lifetime. So that's super tough. So if you can pick daily, go for daily. Like if it makes sense for your, like I would say that, or otherwise weekly. But the more time you have to wait for results, the slower you're going to move. And moving slowly for a startup probably means you will die.

Yeah. Let's end the podcast right here.

You can't see me smiling on the podcast. But I like to say very somber things with a smile. Actually, Michael Kander said this recently. He's like, That's amazing. You say these horrible things and then you smile and then people don't know what happened.

But for those watching on video, they'll be able to see. Okay. So marketing and growth was the same team at Duolingo. When you advise other startups and they're growing and scaling and they have sort of separate teams, talk about that a little bit in terms of like, how do they make sure that they work well together? Don't step on each other's toes. When does one end and the other begin? How do you think about that?

I don't have all the answers for this one, Erik, because I have limited experience. But of course have been approached by a lot of different professionals and been at conferences. I think that the number one point of friction is resources. So I think where most growth teams that I've talked to fail or feel extremely frustrated is when you have a growth team that's in charge of coming up with things to do, but they don't have the people to do them. So then they have to go and be like, hey, can you please, like, can I have an hour of design? Hey, can I please have like, you know, this engineer, can we do this? Can we do that? So everyone needs to be aligned. And that comes ideally, in theory, in a perfect world, from OKRs, meaning that OKRs across teams are aligned. And then of course, at the company level, they're aligned too, so that everyone's working towards the same thing. Because if it is one team's goal to achieve one metric and that the other team is trying to do something else, it's going to be really hard to convince anyone to take time out of the time they could be spending towards their own goals to go help someone else with their goals, unless there's a— a return. And of course, yes, humans are wonderful and we could just share and be nice. But again, it's just in the day-to-day, that's just not what happens because everyone's trying to do their best and they're trying to reach goals. And that's what they were hired to do, frankly. So you can't even get mad at them for it. But I think that would be the number one. I think that behind teams working together is always a great relationship between the humans in that team, specifically the humans leading that team. So there has to be a lot of respect. For one another. And when it comes to brand and growth, for example, yes, I'm a big believer in metrics and growth and optimizing and A/B testing, but there are some intangibles that can't really be A/B tested and can't really be optimized. So, for example, you know, maybe in the short term, talking about Duolingo's mission to bring free language education to the world or making remarks about internationalization wouldn't bring the engagement or the direct users that we would want. And we might be able to do something like show cute pictures of cats that would bring in a lot more engagement in the short term, right? But that's super off-brand. And the brand is the soul. And the soul is what carries this thing through. So you can't just do one and not do the other. So there has to be a conversation among teams. And I saw that a lot in my team as well, which was like our PM, Again, amazing. I think he's leading Duolingo China now. Kai helped me a ton in the team. He was always so focused on optimization because that was his goal that there were lots of points of friction between him and our head of design. And the head of design was just like, I don't care that this is yielding better results. This looks ugly. I'm not going to approve it. Or, I want to do it this way. And Kai would be like, but this is our test and da da da. And he'd be like, I don't care. So there are some things that should be untouchable and that represent something that's beyond the metrics, that's hard to touch. And you can't completely lose sight of that because then you just become this little optimization machine and you build, I don't know, something that doesn't fulfill desires or dreams or missions or that doesn't quite speak to to something else, maybe that the human psyche wants or that, like, humans desire. So, Duolingo is obviously extremely cute and friendly and all, and Duo is this amazing little owl. And there are elements in the design that definitely weren't— aren't there because they were adding to DAUs. They're there because they are adding elements of delight, is what we called it. Like, we need to delight our users, and that's hard to measure. But if it feels delightful, we're gonna keep it. So I think the brand team needs to hold onto that and who are we and what are we here for and make sure that that's being represented across regardless of all the optimizations that need to happen. Because if you also only just lean on your brand, you're also gonna lose.

Do you recommend that the structure that you guys had where either the teams were combined or that there was one leader of both, do you recommend that structure as opposed to having like a VP of marketing and a VP of growth and they're maybe stepping on each other's toes or not sure? who reports to who, or they both report to CEO. But that's— what do you think?

I think that for our— so for our size at the time, the way that we had it was perfect. And I loved it because I had power. I'm kidding. But like, I loved it because I got to make all of these different decisions and I got to own it. And so, and it was great. And from then, like the PR team and social media and anything brand related, like we had the resources we needed to make things happen because We also had these engineers and designers, like most of the time they were working on product, but like we also got to work across things. So it worked perfectly well, but we were, I think we were under 100 employees at the time and shortly after. So along with my decision to leave Duolingo was a realization at the time on my end, honestly, among many other things that I didn't think I was, I was ready to lead Duolingo from, let's say, $300 million to the next 300 million or like a billion users. I didn't think that I had the chops to do that. And I had been able to go from 0.2 to 1, and that was my sweet spot. I think that if I had stayed at Duolingo, so for that next phase, then it makes sense to start, like I said earlier, you start becoming more, having more specialists lead things as you grow because you can afford to and because it makes sense. So then at that point, it does make sense to have a team focused on growth iterations, product, etc. And then, you know, senior management can align at the top whether like everything's on brand and whatnot, but also have a team that's dedicated to brand. That said, you know, I, I didn't focus— so the brand stuff that we did at Duolingo when I was there was very tied to sort of like who we were and how we talked about ourselves and how Duolingo presented itself to the world. but not on spending hundreds of thousands or millions of dollars on ad campaigns to give that brand to the world or to present that brand to the world. And I don't know how much I believe in that because I haven't had the experience of working with that myself. So I know that that's a very limited, like, I don't see God, therefore I'm an atheist way to see the world. And of course we can look at data, but that's just, I can only talk about what I know.

What do you think about the, The concept of a CMO, I meet some people that are like, oh, I don't even believe in having a CMO. What do you think about that construct, but then also what to hire for? If you believe in it, when to hire one and how to think about that?

I think it's funny to say I don't believe in having a CMO. I mean, okay, well then I don't believe in having a CFO. Yeah, I think that having someone who's amazing at what they do and who can lead an organization of people doing that at your organization makes sense. So, and we've identified a couple of core ones over the time of, I don't know, capitalism, CEO, CFO, CMO, et cetera. So I wouldn't question it so much, but yes, like it depends on the size of your org and what you're, and also what you're building. If you're building like a B2B organization and your main engine for growth is coming out of marketing campaigns, that becomes a hugely important role in the organization. If you're building Instagram and the growth is coming from the fact that you have these incredible viral loops built in and you've tapped into something amazing, like from a product perspective, maybe marketing is secondary and then grows over time as that brand grows as well. So, but I think that the chief marketing officer can oversee all of these different things. Now, it also depends Again, because a lot of tech organizations actually have someone like a PM or an engineer step into the growth role, which can also make a lot of sense. Like, I, I am very happy that it worked out for me. And as a marketer, I was able to take this on. And I think that there were a lot of benefits to Duolingo by doing it from that perspective. But I had to learn a lot of math very quickly. Like, I had to understand statistical significance. And why it wasn't okay to do 30 A/B tests of our Christmas email or whatever, our New Year's resolution email. I had to sit down with our director of engineering, Natalie Glanz, who's one of my biggest idols, I would say, still, who's just like, let me explain why. Because I was like, oh my God, I can A/B test everything. I'm going to do 100 different versions. And you need to understand math because so much of growth is numbers, statistics, and thinking that way. So meaning that like every decision you make needs to be like, okay, what is the percent chance that this is going to work out? And like how many, what's the fraction of users we're going to hit? And it's a very mathematical way of thinking. And with regards to paid marketing, we had like a machine learning genius, Hideki, who like stepped in to basically build a little machine learning engine that tested different ad prices for different audiences and targeting different audiences. A marketer can't do that. I'm sorry, marketers. Like, we can come up with a bunch of different slogans and we can go with images and we can test things. But when you start getting super analytical and looking at different datasets, we're no longer equipped to do that unless like maybe marketing school has changed or you went to the school of hard knocks and like had to be the head of growth at Duolingo and had to figure it out. But anyway, so all I'm saying is that I think growth, growth can fall under sort of different organizations depending on what your company looks like. But I personally, from my experience, believe that it is very helpful to have the growth organization and then other marketing brand initiatives under one person who can see what's happening and make sure that one isn't going rogue without the other and that everything's working in alignment. Yeah. What do you think about this question? What would you—

what would your answer be about the CMO?

Yeah.

Oh, beats me. That's why I'm asking.

Okay.

You mentioned A/B tests and I want to segue to that. You have this really great first-round review deep dive on it, but maybe just a high level. What are the common mistakes or misconceptions you see founders making on A/B tests? How do you set up an org that does them right?

There are a lot of mistakes that can be done with A/B tests. I would say the number one is A/B testing too soon. Because when you're really tiny, you just don't have enough users or numbers of page views or whatever to reach statistical significance. And like I said, I myself had to learn what statistical significance was from scratch, and now I can use it in podcasts and sound really smart. But so that means you too, listener, can go Google it right now and you can figure it out. But basically what it means is that like something isn't up to chance. And it's so tempting to be like, okay, I'm going to do a blue button and a green button, and then I'm gonna roll it out to 100 people. And then, oh look, 60 people clicked on green and 40 people clicked on blue. We are going to roll out green and that's the winner, right? It's so nice and it's such a nice story and it makes you feel so accomplished and then you can keep going. But what that does is you're spinning in little circles because you're not actually accomplishing anything unless you know for sure that the result that you got is going to be replicable over and over. That means basically that it's statistically significant, that at a statistical size, like a huge size, this would continue to happen. then you're just guessing and then you might as well just guess. You don't have to do all the work of going through like the whole like A/B testing thing. So it's kind of like a dumber example is if I ask you guys, Jackson and Erik, what's your favorite color? And you both say blue. And I say all men like blue, right? 'Cause all men here like blue. It's like, yeah, but you're 2. So that doesn't really accomplish much. So I think that's problem number one. And it's tough because I think when you're in startup land, you want, you're so, you so want to be A/B testing 'cause it can give you all the answers. Yeah. And you've read so much about A/B testing and growth, but you can't do it early on. And if you do do it, it has to be really strategic and it's probably not going to be as quick as you wanted it to be. So where A/B testing works when you're really tiny is like if you do, for example, 100 people and we give, you know, blue button tonight and green button and 99 click on blue, versus 1, then maybe that's enough. Maybe then you have a real, like an actual thing. And there are calculators for that so that you don't have to just make a guess, but calculators to see basically what's the certainty, like percentage certainty that you can get in that experiment. So that's number 1. And so for really early stage founders, I would say if you're going to use A/B testing, use it from a thought framework. If you're trying to understand, first of all, are you optimizing for the right thing? Is this the biggest lever that you can be pulling? And then what is potentially the biggest result that you can get that can really move the needle and help you go in a direction that makes a lot of sense. One, and then similarly, running a lot of A/B tests at the same time, because again, you have very limited number of users, even if you are already bigger, like Duolingo at the time that I started A/B testing with them. If you're running a lot of A/B tests at the same time, then the number of users that you like that, that are going to see each A/B test and react to them are just going to be smaller and smaller. And if they aren't smaller and smaller, it means that they're overlapping. So then you're making A/B tests dirty, so to speak. Like, remember in science class when you had like the control and then you had the variable? Like, you can't mess with the control. And if like, if, if there's overlap between A/B tests, then nothing's a control anymore and everything's tainted. So you don't get real results. I think going really deep into A/B testing and like getting stuck on one thing and losing sight of everything else that you had on your plate. can also be a mistake. So for example, you make a list of 10 things you could do, and then you work on one of them, it yields results. Then you have a question you need to ask yourself. Okay, this yielded results. Should I run another test and try to get better results? Or should I now go to item 2 on my list of things to do? And you need to look at those 2 options as equals. You can't think like, oh, I'm already on a roll, I'm just going to keep going with this. You can factor that into the ease component, which is like, okay, it's going to be easier to do a version 2 of something I already did because we already have designs and maybe it's easier or whatever. But in terms of how many users can this really get versus like compared to the ease of building it, you need to look at both options at the same time and not get stuck in a little hole of like micro-optimizations. I think relatedly, the biggest mistake is to not take the math seriously enough. And so not hire someone who knows what they're doing when they set up your A/B test and are actually, you know, can verify for you, dear marketer like me, that the data you, you are looking at is clean and real and reliable. Because that's really hard to achieve. Even at Duolingo, we had a bunch of instances where we're like, oh, the data isn't reliable, and then you have to start again. And that's because I think We had some of the best computer scientists in the world. So you need to take that seriously and not just rely on like, oh, well, there's this tool out there and we can do it. Because if you're not plugged in the right way, then you're costing your organization a lot of time. And that's the most expensive thing at a startup.

Thanks for that overview. Speaking of plug, I want to end the interview by you plugging Latitude and the great work that you guys are doing there. Maybe you could also talk about how you're applying some of the things that we talked about today to Latitude. How are you thinking about marketing? How are you thinking about brand building, et cetera?

Well, I did On Deck. I was ODF4. I loved On Deck. I'm a huge On Deck salesperson and spokesperson.

Oh, thank you.

Really. I was so inspired. It was probably the best experience I had in a pandemic year where I wasn't meeting anyone. I met such incredible people and got to to see Erik do— what's it called? The rap?

Yes, freestyle.

Freestyle. No, but I met really incredible people. But that said, you know, I'm saying this because a lot of what we do at Latitude comes from my experience at On Deck, at least with regards to the community aspect of it. So yeah, Erik, thank you. You're amazing. With regards to Latitude, we're doing a number of different things, and there's a piece of it that is On Decky, which is the, you know, community education piece of making starting a successful startup or much more likely to be successful startup a lot more attainable for high potential Latin Americans or people who care about Latin America. I am Latin American and I've seen Latin America stay behind most of my life and it's constantly like up and coming, so exciting, and then crash, up and coming, so excited, and crash. And there's a number of reasons for that. Some are political, some are social and cultural, and some are access to resource. But I think that that can make a big difference. But the way that we're building Latitude is to solve for the question, how can we make successful startups much more common and quick to build in Latin America? So that's one piece of it, but a large piece of what we're doing is also building software to make it a lot easier for founders to succeed. So there's a number of friction points that we identify by talking to hundreds of really high-potential founders from across the whole region, and we understand our problems that either have already existed in Silicon Valley or are unique to Latin America because Latin America is, as I like to say, a clusterfuck. So yeah, so there are problems that are unique and pretty pretty frustrating because things are more complicated and the government and the economy and all these different things. And so we identify those opportunities and start and are building tools. So tech tools for founders. For example, in January, we're launching Latitude Go, which will enable company formation much more quickly, much more cheaply and effectively than what exists today, which is you have to go and talk to a lawyer. It costs you thousands of dollars, lots of thousands of dollars, which to entrepreneurs is often infeasible to start a company like a Brazilian local entity and then have a Delaware LLC or Cayman Island sandwich, which is what we're advising our startups to do because my co-founder Brian, also On Deck student fellow, lost $100 million in his latest transaction because he sold a company last year and he didn't structure it the right way. But a lot of founders are going to make this mistake because they're not going to know someone who's going to say, you need a Cayman LLC, local, da da da, and this is how you do it. And now you can press a button and it's done. So we are finding all of these other friction points that basically make it so that startup founders in Latin America get really screwed because they can't see far enough in advance or because things are super difficult to handle and taking them away so that they can focus on building their startups. And for me and for my co-founders, I think the team, this is meaningful because Tech— well, entrepreneurs are some of the bravest people out there who have the most desire and willingness to solve big problems in the world. I don't believe that Latin American governments will solve big problems in the region, but I believe entrepreneurs can if we can take away dumb barriers from their way. And so hopefully by doing this, we can help create hundreds of thousands of jobs and solve a lot of large-scale problems and plug in Latin America a little bit more into the whole like tech ecosystem in the world and make it much more of a pole because we have incredible talent, incredible problems, a lot of resources. But there are much more, I think I would say even poor, maybe not poor is the right word, but like there are, there are countries out there that are doing much better in being known as tech hubs in the world. Where Latin America is the largest, like the region that occupies the largest latitude in the world and has everything that it takes. And so I think we can make it happen. The time is right. So yeah, we're building this community. We're bringing in the best educators, which are the best operators and founders, not only in Latin America, but in Silicon Valley and around the world. Because it used to be the case that if you're in Brazil, you learn from the best of the best in Brazil. And that's a ceiling. And so we're removing ceilings while also building these products and then also helping founders raise money. We have a small rolling fund of our own, and we might do more on that front to help more people become involved with VC or tech investing, angel investing, and to help bring more diversity into that mix too, because, you know, diversity is a problem in that front all over, but in Latin America it's even worse because we're just behind And that means that if you're a woman or if you are of color or if you're just not rich, you don't have access to any of this. And if we want to build great solutions for the vast majority of the population, we need those people on cap tables too.

No, that's a great note to wrap on. The timing is certainly right. Lots of exciting companies being formed. Lots of employees from those companies can go on and now start companies. But more importantly, the team is right. You and Brian and team. are really exceptional people and really excited to see the work that you guys are doing.

And we made it on deck, obviously.

Oh, thank you for all of your support. Gina, thanks so much for coming on the podcast and dropping a lot of insightful wisdom. If you are building a company in LATAM or LATAM adjacent, highly recommend checking out Latitude. Also have a number of other programs that they have and that they'll be launching. So definitely check that out as well. Gina, thanks so much for coming on the podcast.

Thank you for having me, Erik. It's a pleasure.

Thanks for listening to today's episode. Execs is produced by On Deck, where top talent goes to start companies, find their next roles, or invest in their careers. If you're looking to start a company, uplevel your career, or navigate a career transition, I encourage you to visit beyonddeck.com. See you next time.