#333 Creating an AI-First Data Team | Bilal Zia, Head of Data Science & Analytics at DuoLingo

The biggest success I had was investing in the people, not in ideas. Because I think that order is really important. Ideas don't become successful if you don't have good people in the team. So finding that sweet spot between being able to communicate technical expertise and technical concepts in a simple intuitive way is probably the most valued skill at Duolingo. AI is not going to replace what data scientists do. I believe AI is going to augment what we do. So I think of AI as basically an army of research assistants. So every single data scientist can have access to an army of research assistants. From my perspective, there is still a strong need for a human in the loop.

Welcome to Data Framed. This is Richie. There are a lot of stories about data teams that have found amazing insights that have changed companies tactics and help them make a lot of money or improve the customer experience and involve become heroes. Unfortunately there are even more stories about how data teams had zero impact either because their analyses weren't trusted or because the way they communicated their results wasn't understood by executives. This is a big problem. And on top of this, the role of the data team is changing. Many organizations have a mandate about becoming AI first or AI enabled or some variation on trying to make more use of AI. I want to know how data leaders can approach transforming their teams. Our guest is Bilalzia, a senior director and head of data science and analytics at Duolingo. He moved into data science from economics having had stints at Amazon and the World Bank. Billal is now running a high performing team at a company with a reputation for great use of data and AI. However, when he first took the job, things weren't looking that rosy with pervasive problems with trust and performance. So let's find out how he took his team from zeros to heroes. Hi Bilal, welcome to the show.

Hi Rishi. Nice, nice to see you. Happy to be here.

Yeah, great to speak to you. Now to begin with, I know when you joined Duolingo you told me you inherited an underperforming data team. So I'm curious to be like what had gone wrong?

Yeah, great question. So typically when a new leader joins and inherits a team, they start. They can either start at the ground level or they can build up a team from scratch. What I think I like to joke that what I inherited was not the ground level but the sub basement. And the reason I say that is because there were a few data scientists at Duolingo when I joined but there was no head of data science. So the team was a little bit Rudderless. It's not like the leadership wasn't aware. They were aware of this deficit and they had been looking for a head of data science for a while. Obviously their standards are very high and they were looking for a very specific type of person and they were just looking and looking. What ended up happening was that during this time there was no advocate for these data scientists at the leadership level. And hence there was a big wedge between what the data scientists themselves believed they were working on and the contribution they were making and what leadership believed that they were contributing. As a result, there was basically this lack of trust in what data scientists did, and also a lack of motivation among the data scientists themselves who didn't feel like their work was being appreciated very well. So in that sense, building up that trust with leadership, giving the team the motivation were sort of priorities number one for me when I joined and I had to kind of get up to the ground level first before building the team up from there. Happy to go into sort of more details of that, but that's sort of the high level, what I inherited.

That seems like a common problem when you have data scientists and think, oh, I'm working really hard, and then the manager's like, well, what are the data scientists actually doing? There's often this communication goal and you mentioned that solving the trust problem was the first thing you had to do. Can you talk me through how did you get started solving trust?

Yeah. So when I joined there were essentially a need to understand what are the biggest data related problems that the company is facing, or what are some sort of big problems that company leaders are facing that data can help solve. So it wasn't even, at times, there wasn't even an option that data science or data scientists or economists can help solve a certain problem. So the first thing I did was to actually just generically go to all business leaders, all possible stakeholders, from finance, to the leads of growth, to monetization, to language learning, etc. Just to understand the biggest problems that they're facing, understand the constraints they're facing, and then try and match the types of work that we can do to help solve those problems. So that matching exercise was sort of priority number one for me. And that alone did two things. One, it actually helped me direct resources to the right places. At least I tried. At least the probability of success was higher because I wasn't just doing something that nobody cared about. And the second thing is that it kind of gave the leaders I was talking to something to kind of appreciate that. Look, there's this new leader, he's not just coming in and sort of doing whatever he wants, he's actually listening to us. This is a bottoms up approach and it's a collaborative approach. And I think both of those things were super important. And some of the things we did initially ended up being quite successful. That built the trust battery with leaders and then ultimately we were able to do some big swings as well. And most of them were also successful because they were all informed by what are the biggest constraints and what is the likelihood of success, what is the investment needed. So ultimately I think it was, there's a magic sort of bullet to how to do this, but it was basically just listening to the stakeholders and bringing them along was I think, the key ingredient to building trust.

It sounds so simple when you say it out loud. I just talked to all the people who had business problems, found out what they were found, how data could help them. So it's just matching up like what's needed and what can you actually do to help. Fantastic stuff. So we're going to get into a lot of the details of like how you improve the team, I think. But I want to skip the good bit just for a bit of motivation. Can you talk me through some success stories you've had?

Yeah, for sure. I think the, the first thing that I will say is the biggest success I had was investing in the people, not in ideas because I think that order is really important. Ideas don't become successful if you don't have good people in the team. As I mentioned, I inherited sort of a smattering of data scientists. They were kind of reporting into different parts of the engineering org and their managers are great, they're great engineers, but they're not sort of data science managers and they're not coordinated among each other. So it's kind of, it was kind of acknowledged that this is a stopgap measure. So the first thing was to bring everybody under one umbrella and then to build some team structure. And what that means is building some level of management layers so that junior data scientists have a reporting layer, they have some people to look up to, they have mentors, they have people they can brainstorm with, et cetera. So identifying who are the right folks to be leaders within the org, moving people around, letting a few people go. Because I think people get demotivated by people who are not performing well that actually does have a negative externality to everyone else. So maintaining sort of a high quality bar, hiring people where there was a gap once this initial sort of internal audit was completed and then building a leadership layer. So ultimately where we've ended up is that although I lead the data science org under me, I have people, leaders who lead every single vertical that we work in. And then the org grows under them. So that's how the tree is going to grow and it's scalable and sustainable. And each of those leaders individually are fully invested in their particular pillar. And then they talk to each other as well. So that system, I believe, is much, much better because it gives the blueprint from where impact can actually materialize. So I think the biggest thing I did, where I think I'm the proudest of, is that building that structure from which impact can be generated. Specifically on like to give you an example of where we had the impact. I think the example I've shared in other forums before, I can repeat here is the first thing we did was to pick something that where the likelihood of success was high. So we have, we're a public company, we put out a user forecast. So we have daily active users on a platform. Every quarter we have to forecast what the growth is going to be for the next quarter and for the next year. Underlying this user forecast is a scientific model. It's like a forecasting model. When I joined, there was a model, but there was a lot of discretion that was added on top of this model, partly because there wasn't a lot of trust in the science. And because of this discretion, the forecast was measurably off actuals. So. So it was about a 10% discrepancy between what we forecast, where we would be and where we ended up. And as you can imagine, the CEO didn't really have a lot of trust in this forecast. There would be jokes in their meetings that, oh, this is the forecast, nobody really cares. So the first step was, well, from my perspective, this is a science problem. We can solve this with just doing better science. And so we actually worked really hard to dial up the rigor on the scientific model that generates the forecast. We did back simulations on data from past quarters to see what the results would have been had we applied this methodology and we showed all the leadership in the company those results, which was that we would have been very close to actuals, that sort of built an initial trust battery that, okay, let's try this. And then since then, we have been very accurate in our reporting. We have been within 0 to 3% accuracy for the last four or five quarters. The last quarter was a little bit an oddity. We had some unexpected shocks, but prior to that, we've been between 0 to 3%, which is the sort of gold standard of accuracy. The result of that has been that now all the folks who kind of contributed to changing the scientific model and adding their discretion, they're doing other things. They have other things to do. They're the chief of product, they're the chief engine officer, they're the chief technology officer. They're. They have other things to do that are more important than tinkering with the forecast. And they're busy doing that. They trust the forecast. I think that's been a really big win.

That's pretty amazing. And I have to say, you can tinker around with all the other bits of your business, but if the data science isn't right, your forecasts are off, then, yeah, nothing else is going to work. So I love that he's focused on getting the fundamental analytics right. And then that's the easy way to make, to build trust, is like, get the right answer. Okay, so beyond that, you mentioned that the organizational structure is something you're very proud of. And I kind of feel like in a lot of organizations they're sort of pushing data analysts and data scientists to be more embedded within. Within business units. But you said actually you wanted to bring everyone together so all the data scientists could communicate with each other and have mentors and things like that. Do you want to talk me through some of your decisions around, like, the organizational structure there?

Yeah, I think I see advantages of both of those models. So what we have is a hub and spoke model. So essentially, the data scientists are embedded in individual product teams across the company. So the company itself is divided up into key pillars. So there's the growth pillar that focuses on user growth. There's the monetization pillar that focuses on making money through subscription ads and in app purchases. And then there's the learning pillar that focuses on our content and making on teaching better. And within each of these pillars are several teams that are focused on individual parts of the mandate. And those teams are typically staffed by engineers, product managers, designers, and now data scientists are embedded in that team structure as well. And I think that is super important for me because without context of what the product teams are thinking, what their direction is, what their successes are, we can't be successful. We can't be successful as a sort of ivory tower team sitting in a corner doing our own thing, and then sort of nobody will trust that. Nobody will think we're thought partners. So from my perspective, being embedded in the teams, being included in the team ourselves, thinking that we Are part of the team is really, really important and that has been a key part of how we have been successful and how we've built trust. At the same time, I feel like the hub part of the hub inspoke is really important as well, which is there needs to be centralized reporting within data science because yes, you are a really good thought partner to product engineering design, but you also need sort of folks with the same technical know how as you across pillars to understand how the other pillars, ideas and constraints interact with the work that you're doing because then you can come up with better solutions and at the same time you can brainstorm ideas. So for example, one of the things that our team has pioneered over the last several years is the use of machine learning in product. This started off in monetization where we actually developed a model to predict the likelihood of users subscribing to our product. And that model was very successful monetization. And we have since migrated that same mentality or that same model structure over to user growth where we now have models that predict the likelihood of a user churning so so that we can act before they turn and try and convince them to stay. So that's an example of cross pillar pollenization that's happened because we have a hub structure and I feel that's really important. It's also important for culture, it's important for team spirit. We actually are a really fun bunch. We're not just colleagues, we're friends, we make fun of each other. Our Slack channel is super active. We have a meme wall in our New York City office. We just make fun of ourselves basically. So I feel that that culture aspect is not really possible if you're kind of smattered across different small teams. I feel like a central home is important where you come to be with your people and then you go be with your other types of your people, which is like the teams that you're embedded in in. So I feel like that dual structure is quite important.

Absolutely. Yeah, certainly. I see how if you need the domain knowledge you want to really be solving business problems, then you've got to be close to those commercial teams. But also, yeah, you don't want every data scientist to be inventing stuff on their own. You do need to share with other data scientists and yeah, nerd out a little bit together. You mentioned that this is a happy ending with like all the team being friends, but earlier you said when you took over you had to get rid of some people. And I'm sure it's like very Tough management decisions. But can you talk me through, like how you decide, like how, who do you keep? Who do you get rid of? Who needs to be worked on? Like, what makes, what are the criteria for the people you want to keep?

Yeah, I don't think there's like just sort of hard data on this. A lot of this is subjective. A lot of this is based on feedback from stakeholders and also just talking to the data scientists themselves. So for example, one, so I'll come to the specific question about who to keep and who to let go in a minute. But I think even before that, what was important is how to decide whether someone is capable of managing. So typically in engineering and even in data science, I think there is a tendency that management is just a route to seniority, that everybody's an ic and then at some point in your career you have to start managing because that's the only way to progress in your career. So I am a complete disbeliever in that, in that methodology. I think ICs can grow to very senior levels and remain ICs if that's what they're passionate about. And management should not be seen as kind of, I must do this, you must be passionate about leading people and being a good mentor if you want to do that. So identifying those types of people. So for example, one of the people leaders that I have on my team, she leads our user growth and forecasting and BI teams. Even when I joined, she was actually just an ic. I think she was managing one person and that was also stopgap. But she was informally mentoring a bunch of people. People would just go to her and say, hey, I need help with this. And she would make time, sit down with them, get them through, through the problem, et cetera. So to me that was already kind of, I think she's ready. This is what, this is what she wants to do and this is where she's good at. And now she's thriving. She's, she's one of the best managers we have at Duolingo and not just in data science in terms of how to decide sort of who to let go. I think user feedback, sorry, stakeholder feedback is, is very important and then just the output of their, their work, how long are they taking? What is sort of their mentality around working around constraints. So one thing I've noticed is that people can be the really good data scientists can be very scrappy if our data is not in perfect order, if there are other constraints, if they're facing some other sort of blockages. To their work. They'll find a way, they'll find a way to deliver or they'll be very clear and crisp in their communication. The people who don't do well are the people who complain all the time and they come to me to solve their problems. I think that's fine to do, that's part of my job. But if that's the majority of our discussion, then then I think there is a bit of a problem. There's a lack of effort. There was a bit of that. So just, just sort of letting go of a few people and then using the backfill to bring on people who are significantly, significantly aligned with the way that the rest of the data science org is working. I think that was quite a big game changer for us.

Yeah, certainly I can see how if you're complaining about problems all the time, like data science is just full of endless problems. So yeah, you need to have that kind of level of autonomy and sort of self motivation in order to try and solve stuff or it's not going to work. And I love the idea that just if you are mentoring, informally mentoring colleagues, that's a really good sign that you're kind of ready for management. That's a, seems like a good indicator. You talked about bringing on new people. Are there any particular skills you were looking for when you were hiring new people? Like what do you think are the most important skills for data scientists at the moment?

Yeah, I think the. So this is, this is something that I believe I have changed my mind about over the course of my tenure at Duolingo. So when I first came I thought that if we hire the best technical folks, they're going to be successful at Duolingo. And there's a reason for why I thought that. And the reason I thought that was I came from Amazon. So Amazon is a very different place in their growth trajectory than Duolingo. So Amazon is already pretty well established. They're sort of on the flat part of their S curve, if you want to think about it in business cycle terms. So in order for a data science team or a science team to have an impact, the level of rigor needs to be super duper high. And who are the folks who can do that? The people who have really high technical skills and everything else doesn't matter so much. Like if they're terrible at communicating, you know, that's okay, you know, we're never going to put them in front of a stakeholder, et cetera, which is also fairly standard in some of the big tech Companies at Duolingo, things are very different here. Even our most junior team member gets a chance to present to the CEO. So technical skills will get you through the door, but what will make you successful is how well you can communicate your ideas in simple, intuitive terms. And that's where people were just really struggling. And we had so many candidates who came in. They aced our analytical brainstorm, they aced our technical tests. But then when we asked them to present to us, like do a presentation, a 45 minute presentation on any topic you want to talk about, they completely failed. So either their talk was overly technical that they lost everybody in the room, including data scientists, let alone the product managers who were in the room, or their talk was so simplistic that the data scientists in the room was like, well, I don't really understand the contribution you made here. So finding that sweet spot between being able to communicate technical expertise and technical concepts in a simple, intuitive way is probably the most valued skill at Duolingo. We are very selective in our interview process to find those types of people. It does, it is painful because we have to go through a lot of interviews. Most of the people who come through that pipeline are at stumble at some point, but the people who make it through, obviously people make it through because we, we do hire those people are, have been very successful at Duolingo. So I feel like technical skills for sure, yes. But then being able to communicate that to stakeholders, partner with them closely, bring them along, are equally important priorities for us.

Okay, yeah, certainly finding that communication sweet spot, something I spent my whole career looking for. So, yeah, incredibly important stuff. And it is interesting that I guess the small the company you are, the broader the skill set you need, so you need the technical skills and those communication skills, whereas a larger company, you can sort of get away with being a bit more narrowly focused. Okay, so, you know, we've gone 22 minutes without mentioning AI, which may be, I think, a record for this year. But I'm curious as to whether the rise of generative AI has changed the sort of the profile you're looking for in data scientists.

Let me just speak to it at a company level first. So I think Duolingo has been an adopter of AI even before generative AI became such a, such a big sort of thing in the last couple years. So we've been using AI in the way that we generate our content. We have used AI in tweaking the difficulty of exercises that our user sees based on the exercises that they've just done. And most recently we have a conversation practice tool where one of our world characters has a live conversation with you so you can practice speaking in the language that you're learning. And that's been a pretty big game changer for us because speaking has always been sort of the monkey in terms of learning a language. But now thanks to the advances in LLM, we can actually have an intelligent conversation at a very low cost with our users in their learning journey. So we are definitely pretty high up in terms of the AI adoption for data science in particular, my belief is that AI is not going to replace what data scientists do. I believe that AI is going to augment what we do. So I come from a research background. I spent some time in academia before I switched careers. And one of the really important productivity drivers in academia are research assistants. So these are poor graduate students who we who faculty members just really torture and get a lot out of and pay them no money at all. So I think of AI as basically an army of research assistance assistance. So every single data scientist can have access to an army of research assistants. And these research assistants can go and do sort of simple tasks, even more complicated tasks, but definitely simple tasks that are repetitive, that are predictable and AI can go do this. From my perspective, there is still a strong need for a human in the loop. So there needs to be the supervisor who's sitting on top of these RAs and making sure that they're doing good work, auditing their work and then being the final sort of quality screen before the work actually gets to stakeholders. So that's the way that we have been thinking about AI and that's the way I think it's going to evolve for us over time as well.

Yeah, that's interesting. And certainly, I mean, I've been a long time duolingo user, so yeah, it's always been very good for the sort of reading and writing side of things. But having that actual conversation thing, that's a big game changer compared to just. Well, I'm saying something out loud to myself. So yeah, fantastic technology and innovation there. Going back to your data science examples of being able to automate repetitive tasks. Have you got some specific examples there of things you have automated using AI?

So one of the things that we're working on and we're still playing around with this tech, is being able to generate, querying our database in a simple intuitive way. So typically the way that you will query a database is to write a SQL query. And SQL is a specialized language. Not everybody knows how to write good SQL. And typically Data scientists would get tasked every time. So we have some internal tooling that allows us to query some of our data, and it's pretty good. But what Genai has enabled us is to be able to query that data in a conversational style. So you can basically go to the interface and say, show ME the average DAUs in India from this date to this date, or whatever you want. So literally like you're having a conversation. So that has the potential to unlock a lot more usage from across the company. Anybody who wants access to that, the way that we benefit from that is that those people are not coming to us anymore. So there's an opportunity cost, right? So we save time and then we can spend our time on building the next big thing. So we are investing in that, in that technology, building that interface, and we believe that's going to be a pretty big help for the rest of the company. Another place where we're investing heavily is in our business intelligence capability. So business intelligence, for example, one aspect of that is anomaly detection. So we have metrics that we track and if those metrics are at some point in a day or in a week, they deviate from normal trend, that typically is a red flag that something has gone wrong or something has gone really well. It depends on which side the deviation is. And we want to know, we want to know about what it is and why is it and what can we do about it. All of those questions typically a data scientist would handle, including detecting the anomaly in the first place. So what we have, we are investing in automating is moving away from a world where a data scientist literally has to open 50 dashboards every morning to an AI doing that so it's automated so we can get more sleep in the morning. That's great. And not only that, but the future that we hope we get to is a place where not only does the AI detect the anomaly, but it actually even runs the first set of SQL queries that a data scientist would run anyway, that, okay, well, here's the deviation. Let me run a few data queries to understand what are the key drivers, which segments is this affecting, which geographies is this affecting the most, and then present that data to the data scientist when they log in. So. So it's not just there was an anomaly. Okay, let's figure it out. There was an anomaly, here's what's prevalent, here's the way it showed up in our DAUs, and then it's up to the data scientists to do the next step. And by that time like 50%, 60% of the work's already done. So I feel like that's a, that's a productivity accelerant.

Yeah, I mean I have to. I love that your main metric for success is like, did I get a lie in? Like am I sleeping better? It feels like an incredibly important thing and I guess, yeah, you've, you solve the problems that keep you awake at night. So that's a good decision making process.

Sleep is super important as I'm getting older and realizing.

Absolutely. All right, so yeah, I like the second example about automating monitoring stuff. I think this is something that software teams in general do very well, particularly infrastructure teams. But then it's not really been that pervasive in data science until maybe recently. So yeah, that seems like a very good use case on your, your first use case. This is about getting better self service analytics. I think.

So.

I always thought this is being something that helps like other teams, your commercial teams or less technical teams. But actually you're saying it's really, it's a benefit to the data science team because you're not being bugged all the time to, to jump in and do stuff. I just found the hard part is getting other teams to adopt it and have that confidence to do things themselves. Can you talk me through how you've persuaded all these less technical teams to actually go and do their own analytics?

Yeah, I think it's still a work in progress. It is a good thing that you're pointing your finger at. I think the way that we've done it is we have these sort of pilot sessions where or pilot periods where we invite the more data savvy non technical users into the pilot and they volunteer because they are very interested in this capability and we have them play around with it first and then ask a ton of questions. And that helps us make the prompts better, that helps us sort of make the interface better, et cetera. And we're at that stage right now, so where we have sort of the more data savvy folks helping us make the product better and then ultimately the goal is once they're satisfied with it, then they can help evangelize with their teams and their direct reports. And for a product manager who say is not so tech savvy, they're more likely to listen to their manager who is a PM rather than to a data scientist. So we believe that that way of evangelizing is likely to be more successful.

Okay, so I like this sort of order of who gets to trial these new ideas. So it Starts off with the most technical people and so your top data scientists, whatever, gradually moves on to product managers and then it sort of everyone else. So yeah, interesting flow.

Yeah, I mean I'm a big believer that you can do the best data science in the world and if nobody uses your output when it's useless. So.

Ah, okay, well of course, yeah, you've got zero impact. He's doing fun stuff by yourself. Okay, so actually in terms of like who you have, who the data team has to collaborate with, would you say a product manager is the most important people from your perspective or are there other teams where you think or other roles where you think, okay, we really need to get on the good side.

It's essentially the way that decisions get made at Duolingo or historically have been made are basically the EPD trifecta. So that's engineering, product and design. So you have engineers, product managers and designers. And then now increasingly we are the fourth leg in that stool. So it will be epdd. So engineering, product design and data science, Many, many teams across these pillars we are already considered an integral fourth pillar, a fourth stool, a fourth leg in the stool. And in some ways the way to summarize the way I think about the role of a data scientist is that a data scientist is basically a product manager with a technical hat. They think like a product manager, they think like as owners of the product, they're invested in its success, they're all constantly problem solving and they're bringing data to come up with a better hypothesis for what to do next.

That's pretty cool. I've not really thought of data scientists being a type of product manager. Suppose you are a data scientist, you want to get more product focused. What's step one in doing that?

Just investing more time in the context so I can give you my own example. So when I joined actually I really do credit our CFO Matt Scarupa. When I joined. So I joined from a culture where basically it's you join and you hit go and you go, you don't, you have no time to for better or for worse. It's just a very sort of fast paced culture. I really enjoyed it when I was there. When I joined Duolingo, the first piece of advice that he gave me was Bilal, you've just joined. We have a very rigorous selection process. Obviously we've been looking for a head for two years and now finally we have you. So I don't have any expectations from you. For the first month I want you to sit and be a sponge and be and get context of what Duolingo does, what our product teams work on, how they work on things. What have we tried in the past? Why have things failed if they have failed and why have things succeeded? Build that mental model and then augment it. And I really kind of. That advice really resonated with me. I kind of passed that along to everybody who joins the company and talks to me, because I feel that's really important. Without the right context, you can do the best data science in the world, you can build the best economic models in the world, but they're just not gonna. You're just gonna be shooting in the dark a and then probably misfiring most of the time. And if you have the context right, you still get a for effort, even if your solution is 80%.

I do like the idea of just making sure that new hires just get to soak up a bit about, like, company culture. What is needed? What are the problems you're actually trying to solve? So having that, like, thorough onboarding just seem like a good way to set people up for success.

Yeah. We now invest a lot in preparing the onboarding documentation. So we have a whole, basically a masterdoc that provides for each pillar, depending on which pillar you're joining, a history of what teams have been working on, what types of problems, links to the experiment dashboard, the types of experiments they run, a list of people that the new hire should meet in their first day, their first week, the first month, and just to give them a head start to start developing that context and those relationships.

That seems incredibly useful. All right, so I'd love to zoom out a bit to the company level. So there's a lot of companies where, I mean, there's a few buzzwords being thrown about, like they're trying to become AI first or AI ready or AI enabled. What does that mean for you as a manager at Duolingo?

I think the way that we think about AI is how can it actually enhance our productivity? So a lot of, I believe a lot of hype around AI, at least initially, has been around cost cutting. How can AI be cost cutting? I think that's important, but I think more important or more sort of exciting for us as a customer company, is how can it actually make our product better? So the example that I gave you on the video call feature, where you can actually have a conversation, is an example where it's enhancing our product. It's unlocking a capability for providing speech practice which never existed before. I mean, the only way to do that before AI was for us to actually hire tutors, like human tutors, and interact and for them to interact with users. And when we have 50 million DAUs, that's just not possible. This is not feasible. So AI has unlocked that. So our, our stance towards AI is we are invested. We are invested in making AI improve our product. And that's where we see the promise the most.

I love that because, yeah, certainly, like, cost cutting is important, but it's like it's, it's only your CFO is going to be really excited about it. Whereas actually if you're trying to improve the customer experience, you know, you're making, well, 50 million people happy.

And like, if we sit and do nothing, AI costs are going to go down. Right, because of everything that's happening in the industry. So the costs have been coming down because of new models coming out, things getting cheaper, the foundational labs making better and cheaper models that go faster with less compute. So the, the costs are going to come down on their own through market dynamics. So even if we do nothing, that cost component is going to help us.

Okay, and do you have any advice for other data leaders on like, what they should be doing to encourage their data teams to have more use of AI?

I think definitely creating an environment where they have the incentives to try different AI tools and not penalize when there is a productivity drop. So a lot of times we use AI tools and they really suck. And then, and then we evangelize that learning among everyone else and then we, or we sort of trial an error more to see what works, what doesn't work, but just making room in the team to do that, and for that not to be punitive, I think is really important because that's how you discover what's useful. So an example for how it's been really helpful for me is custom GPTs. So I have a custom GPT that I call something like a leadership ally. It's been a game changer for me because it's kind of like having an executive coach. It's not perfect. A lot of times it's very agreeable, which I don't like. And I've tried to prompt it to be very critical and it does at times a really good job. But it's been very, very helpful in the way I develop communication with the rest of the company, the way I communicate with my direct reports, the way I communicate with the team, and I find that super duper helpful. It's been a very big game changer for me.

I do like the idea of having something to just second guess or you just, or at least critique your decisions. But yeah, certainly AI being too agreeable. It's, it's a common problem. It's like I want you to tell me things that are wrong. It's. Oh yeah, you've done a great job here. Yeah.

Okay.

I think like one of the big fears is around job security with AI and like are you going to automate yourself away? I know you talked a bit about psychological safety. Do you have any tips on how you can ensure that kind of. That your team feel comfortable around that?

So it's an interesting question around job security. So if you believe my hypothesis about AI being sort of augmenting, not replacing productivity, then you can imagine that the demand for data scientists is actually going to go up rather than down because if we can do more than the out sort of the marginal cost of output that a data scientist produces goes down because each data scientist can do more. And if that's the case, well, every company is going to want more data scientists because they can do more with less, in my view. I believe, I believe in that hypothesis. So giving the teams the right incentives to play around with AI, get comfortable with it, use it in their work is really, really important. And for data scientists to be open minded about it, not close minded about it, because AI is definitely here, it's not going away, it's the productivity enhancer. They should think about it that way and they should find ways for, in which it's super helpful for them.

Absolutely, yes. I feel like in most companies it's not like there's a shortage of data science problems to be solved. So productivity enhances, productivity boosters. They're going to be incredibly useful. I don't think. Well, there may be some companies where they're like, okay, data type is just too expensive, we'll get rid of them. But if you're showing your value, you've got that trust with management, then it ought to be an overall benefit, I think. Okay, wonderful. All right, so can you talk with you. What are you most excited about in the world of data and AI?

Yeah, it's a good question. So we do a lot of. So this is aspirational. I believe you're asking an aspirational question. I believe the future, what I'm going to discuss is something that I believe is the future of how we think about sort of experiments and testing and a B testing in particular. So we are very big into AB testing. We run hundreds of A B tests every quarter at Duolingo. Every single product feature that gets shipped out is a B tested first. And this is true for a lot of companies, not just duolingo. Now one of the perils of a B testing is that there typically is a blast radius around a b test, around 50%. I think on average a B tests are not successful. They don't get, they don't get launched. So our capability of generating great hypotheses is not very good. So we test things and then, and then half the time we're wrong, it doesn't work out. And what happens with, with those things that don't work out is at times things actually go get worse that we, we do more harm than good in some experiments. So there's a blast radius. What if there was a way for us to actually pre assess the blast radius and not do experiments on humans or on our users that would actually not benefit them? So I believe there's a lot of research going on in this, but there is definitely a world where we will be doing synthetic AB testing. So instead of real users, we can actually use LLMs to mimic the behavior and decision making of ll actual humans and then be able to run hundreds of A B tests, not in a quarter, but overnight. Now there's a lot of problems to solve around this. There's problems on how best you mimic human decision making, there's problems of cost. But I believe this is the future and I'm pretty excited about it.

That's a pretty radical idea. The idea that you've got eight or bots instead of real people, you can run your a B test against those and see what performs better. So you're doing optimization in a safe way before you exposing your users to sketching new features. All right, I really like that as an idea. Yeah, I guess, fingers crossed, we can make it happen or someone can make it happen. Finally, I always want you people to follow. Whose work are you most excited about.

At the moment along the lines of this type of sort of synthetic A B testing? There is a team at Amazon where my former team actually is working on this and then they're collaborating with some really top academics. So there is an economist at mit, Victor Cherno Sukov, who I he's a very famous econometrician. I follow his work and he's involved in some of this. They just released a paper, I believe it's called Agentic Economic Modeling. So that's basically an economics based framework around thinking about synthetic A B testing. So I do follow his work quite a bit. So yeah, that would be something that I'M or at least a person that I'm following currently.

Okay, wonderful. Yeah. I mean, it just seems such a fascinating topic, I think. Definitely something to dive into in more depth. Maybe a future Data Framed episode. Wonderful. All right. Thank you so much for your time, Billel.

Thank you. Sa.