#102 How an Always-Learning Culture Drives Innovation at Shopify

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey. On datacamp.com.

You're listening to Data Framed, a podcast by Data Camp. In this show, you'll hear all the latest trends and insights in data science. Whether you're just getting started in your data career or you're a data leader looking to scale data driven decisions in your organization, join us for in depth discussions with data and analytics leaders at the forefront of the data revolution. Let's dive right in.

Hello everyone, this is Adele, Data science educator and evangelist at datacamp. One thing we keep thinking about here at datacamp is how important a culture of continuous learning is for data teams. Data science is still relatively nascent compared to other technology disciplines like software engineering, and too often now we see new frameworks, new tools and new ways of work for data teams. This definitely requires a culture of continuous learning for data scientists and this is why I'm so excited for today's guest. Ella Hilal is the VP of Data Science and Engineering at Shopify's Commercial and Service Lines Division. She is a well seasoned data leader with an extensive resume that I'm not doing justice with this short introduction. She's led a variety of projects and is an expert in areas such as data analysis, machine learning, autonomous systems, IoT, to name a few. She's also an incredible learning advocate for the data scientists that she leads. Throughout the episode we speak about her experience leading data teams at Shopify, how data scientists can develop a continuous learning mindset, how data leaders can create space for innovation within their teams, some of the use cases that she's worked on and her biggest learnings from them, and much more. If you enjoyed this episode, make sure to rate, comment and subscribe, but only if you like them. Now onto today's episode.

Ella, it's great to have you on the show.

Thank you. I'm really excited to be here.

Awesome. So I'm excited to discuss with you the data science powering Shopify. How you approach an always learning mindset, how you lead data teams and More. But before we begin, I'd love to talk about your background and how you got to where you are today. So can you briefly walk us through your journey and how you joined Shopify?

So I would start with I'm a girl from the Middle East. I actually went to university in Cairo and I studied computer engineering. Then I traveled to do my Master's. I did my master's jointly between Cagrau and Ulm University in Germany, which was amazing. I got to learn a lot. I did take some courses from Stuttgart University. I got to visit the different campuses around Germany and then I went. At the time I had full scholarships from Fulbright, from the ADE and in Canada too from OGS and Insert. And I ended up landing in University of Waterloo where I I studied machine learning in AI and then I graduated. And then I'm not going to take you through my whole career, but maybe I'll give you a couple of highlights. I have a PhD in patent analysis and machine intelligence. I started my career as a Java developer because when I started data science was not a thing at the time. And then from there I started leading innovation teams and then started data teams and then grew into leading the data science organization in a company called Intelligent Mechatronic Systems. Then I moved into Shopify as the director of data for plus, which is large size merchants and international. Plus is like a lot of our big merchants, like Tesla, General Electric, some of the Kardashians, like name it, anybody who is anybody is on there. We have lots of very amazing, very talented merchants building their own brands and international. We started with the mission of making Shopify a perfect market fit for all the markets we're in. We're Already, we're in 175 markets, but we were started with the intention of making making it feel like solving local needs, not just a platform, a global platform that operates regardless of the needs of the merchants. And from there grew into leading the growth and revenue organizations. And now I'm the VP of Data Science, heading all the commercial and service data science teams.

And given your extensive experience as a data leader at Shopify, you know, one thing that I've seen you speak about, and this definitely requires it to be able to succeed in such a role, is developing an always learning mindset, which is really what I want to center today's episode about. So I'd love to set the stage for our conversation today on how you define an always learning mindset and why you think it's so important for data scientists to progress within their careers.

I think this is the most important superpower data scientists can have. Like, I talk to a lot of leaders in data science world and they talk, oh my goodness, we need somebody with a PhD and we need somebody with a master's or X or Y or Z. And I was like, don't get me wrong, I do have a master's, I do have a PhD. But I don't think that's what makes a good data scientist. I actually think the good data scientist is the one who has this learner's mindset and learner's mindset, I define it, is the person who is able to go back and learn, doesn't get stuck in what they know. They are actually able to continue collecting additional tools, additional formats of thinking, of data, philosophies, mindsets, frameworks, along their journey to add to their toolbox. The data science craft in general is evolving rapidly, and it's relatively in early stage compared to engineering and other crafts who've been there for like many, many more years. And because of that, things are evolving fast, frameworks are evolving fast, and we're coming with new techniques and new approaches all the time. The trick is not knowing the latest all the time. But the trick is being able to learn new techniques when the right questions and the right setups come. So it's not about the shiny new thing, it's about picking the right tool out of your toolbox and if it's not there, going and finding it and getting it and adding it and learning how to use it.

That's really great. I'm excited to expand with you the methodology for learning here that you've acquired along the years. One thing that you mentioned upon is one, data science is not relatively mature yet compared to other fields. But also data science is inherently multidisciplinary. You know, data scientists are required to blend two broad skill sets to be able to deliver value. One of them is business acumen, right? Knowing the product that you're working on, having communication skills, being able to work with collaborators on business problems, but also technical skills. Nuts and bolts of data science, as we say. So starting off maybe with a technical skill set, because that's arguably the more comfortable one to grow in as a data scientist. A lot of the growth data scientists have on the technical side comes from actually learning new tools and experiments on the job, as you said. However, given the importance of delivering value in the near term, how should data scientists maneuver the trade off between applying tried and tested techniques to solve problems and learning and experimenting with new tools that May not pan out to deliver short term value. Yeah.

So I will divide this answer into two. I always believe in focusing on impact and by focusing on impact you can always iterate like I believe in incremental shipping. So what you can start with is let's say a simple thing you're asked to do forecasting for like a top line metric. You can go with the latest, coolest and fanciest paper that was published about like this neural network that allows you to optimize a hyper model with like many parameters and then like takes that to do something with like some form of like hypertuning for some regression or whatever. You can go with these some very complicated techniques multilayered right away. But don't get me wrong, yes, you learned something cool. But did you really solve the business problem? Did you really know how to use it effectively? Did you know effectively your baseline? I don't think so. I think the right way to go about it is to start with your simplest. You know what, let's line fit and from and then take it one more step further. Let's apply some, some linear regression and maybe you know what, let's do some logistic regression maybe then we. And as you iterate, you understand the progress and you understand your data, you understand your different parameters, you understand your levers that you're pulling. And then as you iterate, you're actually finding more and more and learning more and more and have better understanding why you're leveraging and using this. One of the biggest mistakes that I see data scientists do is to try to be at the cutting edge of technology. They run to the shiniest thing right away. And the problem is the Chinese thing doesn't mean that this is the most important or practical thing. To be effective and to be successful and to reach this mastery in your craft, you need to understand what exact tool to pull out of your toolbox to solve the problem in the most impactful way. It's not the fanciest Chinese tool. It's the appropriate right sized tool. And to do this you need to build this sense of iteration, incremental shipping. As you iterate over time you get better and accordingly you can use more sophisticated techniques. The more sophisticated techniques actually sometimes blinds you from why it's operating this way because it's a black box and like you spend a lot of time trying to throw things at it. But the truth is you're throwing things at the wall versus you're really trying to understand what levers are you pulling at the end of the Day. Any machine learning model is a, literally a line fitting in high or like hyperplane fitting within two between multiple dimensions. That's it's, it's linear math, guys, it's math. It's not, it's not rocket science, it's math. And if you understand that then any new technique is not shiny. You need to understand the underlying math to choose and to understand the underlying math, you can't start with the most complicated equation, you need to start from the beginning to progress through it. So with that mindset I think you tend to iterate and learn. The second thing that I always do with my teams is like we use something called BLT time or like hack days or like whether it's paper gramming time, which is a great way to learn new things, data digest which places where people can present their work and teach each other or hack days where you can get to experiment with new things. So like you always need to have some like space in your scheduled space to get to pick new things and experiment and learn. But on the job, on the day to day job, you can also learn through iteration and through pushing the boundary and also through experimentation. But don't start with the thing that you cannot debug and understand the why it's working or not. Start simple and iterate.

So there's two frameworks at play here. The first framework is constant iteration and starting off from simple solutions and avoiding that shiny toy temptation. Because I think a lot of data scientists fall into that resume driven development pathway. And the second framework is also as a leader creating that space for teams to share knowledge and experiment with new tools that may or may not be shiny and kind of present their work as well throughout the way. Is that correct?

Yeah, totally. And regardless, shiny or not, the understand, if you understand the why underneath you understand the distribution of your data sets, you can iterate, you can have even better ways to enhance existing algorithms or even new algorithms.

Now for the business acumen side, I think that's arguably a more challenging skill set for data scientists because it blends communication skills, collaboration, product sense and more. And it's not something that you learn necessarily in a data science education and it's not something a technically minded person would be geared towards potentially. What are frameworks and mental models and similar here as well mechanisms within a team that you find useful to improve that skill set continuously as well.

I love this question. I can't tell you how much I love this question. So I tell you the biggest thing and the most important thing that I keep repeating. Is like, as data scientists, we need to focus on the outcome, not the output. And I know the sentence is very simple, but it's so true. We a lot of times focus about shipping an algorithm, but we don't ship the business impact. We don't focus on the business impact, which is the outcome. We focus on shipping the algorithm. So for us to get ourselves to tie to the business impact, I think one of the key tools that like I recommend for everybody to use and I actually reference a lot is the five whys. You need to understand why we want to do this and you want to actually debate it from a human level. So for example, if I want to tell you, build me a recommendation engine, that's the sentence that the PM product manager can come and say. It's like the question is why? And then you can say, we need to recommend themes for merchants themes, which is like what's template to you. So then another why would we want to save time? Da da, da. Then as you continue the conversation, you realize that when merchants come in to start their stores, one of their highest friction points is actually choosing what theme to use for the right business. They want to make it unique, but they want to make it useful. They want to make it appropriate to the product that they're selling, but they still want to put their flare on it. So being like this intelligent partner, like this automated intelligent recommendation assistant type of algorithm with them tends to save them a lot of time and actually become a sounding board too. That has a real impact for merchants. And when you understand that, you can actually start with, you know what, you actually don't need the recommendation engineer. You maybe let's start with the ranking and take it from there. And then as you get the ranking, maybe the next iteration will be a full recommendation engine, right? Like, so you can iterate over time knowing the outcome that you're trying to drive for and using your skill sets and this massive toolbox that you've been building from our step one to be able to pull the right thing versus acting on a specific ask. And the business acumen is built by being curious. There's no other hacks. I can give you a ton of frameworks, but all of them are founded in us asking questions and asking to understand the drivers, keeping an eye on the outcome, not on just what we're shipping. And that makes a big difference. And also you're going to see massive engagement difference with their counterparts. So if you're working with product managers, get to see them interact with you differently. If you're working with engineers with even like sales reps or anybody. At the end of the day you have also shared language and this is really important. Regardless technical or non technical business problems or business acumen, business problems are common and shared by all the crafts working in a certain group or company. Now your language changes from like a data science craft language to a business common language shared across. So the bond, the connection, the alignment becomes much more amplified and faster.

That's really great and I love that. What's nice about the framework that you laid out is that by breaking down a business problem into its multiple components of parts, like with the five whys for example, you're able to also break down a technical solution into its component parts and be iterative from there totally.

And also you get to understand the drivers, not just the translation of it by a pm. So the PM heard something and came back. I literally was in a conversation yesterday and somebody came and was like I want a neural network. And it's like why? And then when we started talking about it it's like yes, he needs to do a classification. Maybe neural network is not the best choice for the data set because again any machine learning model has its underlying statistics. So like maybe we're over complicating while we just it's a linear data, we just need something much simpler. So it's, it's all about this conversation to understand. To understand also what assumptions are made when you're discussing because you're talking about them, the human, the usage. Like when for example, if we go back to the example of the recommendation engine for the themes, it also makes an assumption as you ask these wise you get the assumption on when is the merchant going to use it, at what phase of their journey that they're doing it early enough. Right. Like they're not getting used to Shopify yet you get to understand that they, they don't maybe have a full theme for their business. So maybe that actually gets you another idea of a different ranking or recommendation or like whatever additional tool that you can provide them in a separate step that can make this step easier for them. Right. Like it can give you this sense of the, the merchant journey and the information around it and accordingly you can build these different components and get to see just that product even can be an ecosystem of products around it.

That's really awesome. And we had on the podcast last year Shafri Bahar VP of Data Science at Go Check, which is also like a highly data mature organization. And one thing he mentions is it's really Important to embed data scientists in this different business teams simply because it enables that common business language and it enables that skin in the game for data scientists on the solutions that they're developing. Do you share that worldview and how has that been effective for you at Shopify?

The data science team in Shopify is a centralized craft, but we work with embedded teams. So what does that mean? That each team is embedded within their own organization. The reason is they need to be close to the business problem. Data science cannot be behind a wall where you throw things over with questions and expect like proper answers to come on the other side. Because even basic questions has an assumption. So for example, if I tell you, how is the buyers doing on our merchants website? So buyers are the customers buying from our merchants, merchants are our own customers. Right. Very simple question. What defines a buyer? Is it the one that comes to checkout? Is it the one that just goes in to browse? Is it a session that is starting? Somebody just hops in and leaves. What defines a buyer? So there is these discussions and these understanding and like being close to the problem space helps build number one, this better mindset and understanding of how the things work that enables data scientists to do their jobs better create the common language between the different groups as well as creates a further, much bigger curiosity about how the product itself works.

That's really great. I love that. And really, I think this marks a great segue to how data teams at Shopify are leveling up their skill sets and becoming, you know, and adopting this learning, always learning mindset that you talked about, connecting back to the trade off maybe between short term priorities and longer term innovation investments. How do you approach that trade off as a leader in your own teams? And how do you create time for your team to experiment with new skills? And you want to maybe walk us a bit into more detail of what these programs look like.

Yeah, that's, that's great. And there are multiple different programs. So we have something. I love this thing. I had this done many years ago, like seven or eight years now. And I've been using it ever since for every single team I led called mini Sprints. So it's the idea is a similar to the idea of hack days where like hey, hack days, everybody come and build. But you don't need to always invoke massive scale hack days. Somebody on the team has an idea and that we believe in it. Like let's say, you know what, I think I can make this 20% better. I just need a couple of days. Amazing. We can invoke mini sprint. That person now invoked a mini sprint. So it's not that they by themselves will do it. You can collect people from different groups and say like you guys, four people. There's this vision, go build a mini sprint, experiment with it and come back. So the investment is small. The investment is two to three days. Sometimes I do it all the way to up to a week. But usually it's like, it's like a spike. It's a small. But the value of it is like it's cross teams. It doesn't have to be specific teams. It also creates high bond between the different groups that are working with it. But also it creates a space for quick innovation and experimentation to prove a concept like, similar to the idea of spikes, but instead of pre planned within the same group, it's across the groups and it's invoked by either an important business need or a question. So that, that allows a lot of like us experimenting fast and failing fast and feeling forward. Right? Worst case scenario here, this team, these four or five people build a bond and we usually diversify the people. So this way we continue building bonds and connections between the different teams. That's the worst case scenario. Best case scenario, we'll learn something very useful, whether a positive learning or negative learning, which is like learning of stuff that didn't work or learning stuff that worked. So that's, that's a great way to do that where it fosters like experimentation and innovation. But we also have a very specific cycle called what we call the vault projects, which is proposal, prototype. And then we go into the build. The build is we building for a long term. We're building and optimizing and being able to build like robust, reliable engineering systems. But in the prototype this is the phase where we. It's a normal cycle, normal sprint or two. But the in the prototyping is what you're standing up fast. Unlock the business and by having naming. So what I shared with you is two techniques for experimentation as well as differentiation between fast experiments and long term builds. Why am I saying that? Because having naming for both, having phases for both, intelligently calling them both allows us to focus on the trade offs that we're making. The problem is when you're building something fast and putting it aside and forgetting that it's fast hacking, this is where technical debts arise for you to solve. For that you need to have words and names for it and you need to have intentionality. You need to differentiate between the quality of the output of these two phases and Accordingly, if you have an output from a prototype, the expectation that it's in a beta, if you're lucky, if it's not an alpha, but the output of a productionized cycle or built cycle, it's a fully productionized system. So it's more robust, more reliability. So by having this, by having the internship, when you're building your roadmap, you're clearly calling out what phase this is in, creates the space, the intentionality for you to ship quickly, to unlock the business, but also plan for the longer term and iterate for the longer term. Maybe one thing that I would like to touch on on here, because I know that a lot of data scientists suffer from this is ad hoc questions that tend to eat most of people's time. I think there is a big opportunity miss. When we take the adult question, we hate them and that's okay. I know they are disruptive, but then we just walk away. But the truth is the ad hoc question came because there is a system that is missing or a system that is broken. If we pause, reflect, maybe do an RCA root cause analysis, like sit with the group, it's like, why do you think we're getting these questions? What is missing? You might find specific reporting that is missing, you might find specific tooling that is missing. And accordingly, you can move these fast type of questions into system building with an objective to reduce these. If you do this effectively, you might like, I have cases where we were very successful to reduce adult questions by 70 or 80%.

Wow, that's really awesome. And I want to kind of unpack a lot of these different initiatives and programs you have set in place. Maybe starting off with the mini sprints, how do you ensure here in this situation as a leader that the time spent by the team on mini sprints. Right. You mentioned that the worst case scenario is that people bond. But how do you balance between the absolute objectives that we need to, you know, land this quarter in this space for the mini sprints that we need to have within the quarter. So what's, what's your, what's the barometer that you use?

I love that question. And that's part also of the mini sprint like that I have, which is whenever you have an idea, it's not randomly you have an idea and you just go build it. You need to bubble it up and bounce it on your leads to do the trade offs. And if that's the case, it's an investment that the leads do because you never run it by yourself. You run it with couple more people so it's with intentionality and usually because we're making it visible, it's not side of this work. We get to know the beginning of the mini sprint because there's a raw around it like hey, we're starting a mini sprint, da da da da. And then at the end of it, people send a summary of the mini sprint. So accordingly it creates a sense of ownership and accountability. So people are not just running randomly doing this because it's not seen, because it's visible, people want to do good work and because it's communicated, people are intentional about is it worth it or not?

And maybe touching upon the last element of your answer. System building and ad hoc requests. I know this is something data scientists really hate. How does that you mentioned here that definitely like ad hoc requests create that, create that connection to understand what are the systems that we need to provide, the tools that we need to provide. Walk us through maybe how self service analytics can solve a lot of these problems, right. Of ad hoc requests and maybe walk us through some of examples of how you more in detail of how you were able to drop down that ad hoc request by 70%. Because I know that there's a lot of data leaders listening to the show who want to learn that secret.

Totally happy to. So the fact is like ad hoc questions are not coming without a real business need. And if they are, we should actually say no, no, thank you, we have other more important stuff to do. But if they're coming in for the business need, let's look at what is recurring and what can we see. So for example, one thing that came in the team, the plus data team at the time was very annoyed with it is the fact that every time we were doing like some email marketing back in the day, hey, we needed to get like a list of emails and this is a pii. So it needs to go through data and we need to make sure that we do any cross checks to make sure that we're respecting like people who are opting in and opting out and da da da da. So it was. And at the time because the system was fragmented between like Shopify and the Plus Merchant, da da da da. We had to do many, many steps manually. This is, this is a problem that takes a good two to three hours of every time a question like this comes the problem. Also it used to come within like they already built the whole campaign and now they need it, they need it in the next 24 hours, give it to me right now type of thing. So if you look at this, this is definitely a candidate for systemization. First of all, requests need to have X amount of business days or turnaround unless there's exceptions. Number two, a lot of these pieces in the system where like there was manual validations and stuff like that, that all can be automatic automated. So by doing this and creating right reporting with right alerts and right checks, we just build the system. And that now is not as dreadful or not like doesn't need as much involvement of a data scientist anymore for every time we're sending like an email for our massive scale of merchants. So that's a simple example. Like you can all always been complaining about like oh my goodness, these questions come, but just seeing the pattern and each of them come doesn't come in with the same data poll. It comes like, oh, we're doing this campaign, we need support. Data support. Oh, we're doing this new campaign, we need data support. Same thing about like what's happening in our funnel. Very simple question. You can again every time answer the question or. Exactly. Very, very complex to answer. But if you do it enough times, you get to see like 70% of the answer is actually systematic charts that you're looking for. So you can then go build a reporting suite. And I use the word suite, I don't say reporting dash. The reason I'm saying suite is because you need to think about what type of dashboards are you building and how do they interact with each other. If you think about your reporting as a data product that will set you up for success. The reason I'm saying this is because when you think about it as a product, you think about user experience, you think about navigation, you think about the uptime, think about a lot of things that actually big part of the reason dashboards get abandoned in the dark hole of dashboards is because we don't think about these things. We create a lot of one off dashboards because it's easy, but don't create a navigation between them. We don't make sure that these answer cohesive, comprehensive questions. We just like each of them answers a random piece of the puzzle. But how do we navigate this? Now we're needing to pull a data scientist to do it and data science hates doing that because it's dreadful work, not cool work. So if you step back and think about it from a data product side, it becomes now a data product and it becomes now with all the user experience that comes with it and it's up, it's running, it's easy to navigate and works a lot Better. So again, this is how I solved for a lot of these things. I stepped back, looked at the ad hoc questions that coming in and every time we see a good collection, we try to systemize by solving, solving for the underlying root cause.

That's really great. And the keyword here that you mentioned is product, right, Data product. And I think that when you develop a dashboard or reporting suite, as you mentioned, having that attention to user experience and how your dashboard is going to get consumed is something that I think a lot of data scientists miss. Necessarily because it is at the end of day a digital product that people will consume. It needs to have the same type of experiences or expectations that people have out of digital products products.

I do agree with that. Like again the whole idea is think about your own experience from data. Like if you're a data scientist using, I don't know, Google Analytics or you're using your analytics on your Twitter or any of the tools that you use, what do you want to see and what makes sense to you? And if you start seeing the themes of experiences that you enjoy and start bringing these into the dashboards you build and bringing these into the tooling that you build, it becomes like again, it's easier to adopt and more enjoyable to use for business stakeholders and accordingly less pull on your attention.

So we definitely talked about how creating these systems for the wider organization helps out in one, reducing the workload for data teams but also helps out accelerating data driven decisions, improving business outcomes across the organization. Right. And automates a lot of different tasks. How much does data culture and organizational data literacy for non technical stakeholders play a role in creating consumers for the data team's outputs?

That's a great question. I'll tell you it makes a huge difference. However most organizations like when you start the group and interactions is similar to any relationship, right? Like you don't start with everybody knowing how exactly to work with each other perfectly even if they're coming from data driven previous role or organization or what have you doesn't mean that like it's just gonna click. So by having high intentionality and showing value repetitive it tends to elevate up the data understanding. So we do have in Shopify like many courses for non data scientists to up level on data science. So like how do you understand charts or like how do you write SQL if you're interested or any of that. But I think the key, the real key that makes a pivotal change is having the right level of conversation and showing value. If you're talking with complicated equations, you lose People if you're talking with the language and that goes back to the business acumen pieces, you go back to talk about the business problems, which is a common shared language regardless of the craft, people tend to listen more and tend to understand more. It's on us as experts in our domains to be able to play this translator role where we talk from a business perspective and doesn't mean that we take it down or not like talk fancy, but like it means that we talk with what really matters, which is the business and the impact on the customers, the consumers. Like I don't think talking with very high precision when it comes to the data science crafts serves us better when nobody understands. I think think being understood is more important than being precise. When you're talking like if you're Talking about your F1 score and your sensitivity and your precision and your false positives and your true. Like if you're talking about all of these things, all of these, like we all use them in day to day life when we're talking to each other. But if you talk to a business stakeholder and you're talking about all of this and all of that just like doesn't register at all in their head, then you're both on the losing side of this conversation. But if you simplify it to what really matters and they are able to action your learnings because they understood it, you're both on the winning side of this. So it's really important to keep that in mind.

I completely agree on the last point and I think it's extremely detrimental for data teams because if this happens in front of an executive, for example, what you're going to have is loss of executive trust in the data team and less investment in the data team's longer term output and work production 100%.

So I will tell you something funny, I actually did see that. So for example, a data scientist runs an experiment and the experiment is set up as an E B test. But of course anything is set up has some form of caveats. So the data scientist comes in and sharing the insight with SLT and this is a true story. I'm just like abstracting and the data scientist wants to be so precise in the words that they're using. So they went in and the experiment was like had a positive impact.

Impact.

Their intention getting into this meeting is to advocate for like rolling this experiment out to everybody. And they went in and to be precise and trying to be unbiased. They did so much listing of the caveats that what happened is people in this meeting just assume that this experiment is useless. And they like, they struck it out. Although it was rigorous, it was done right, there was proper significance, everything was right. It's just like again, this data scientist got in their head so much and they talked so much with the data science language that what happened is the opposite outcome of their intention. When they went into this meeting, that's.

A great story and probably would have been better off, it said, hey, I ran this experiment, this is what we should do. This is the expected outcome. And if you want to read the appendix, here's the appendix.

Exactly. Or even if you want to say caveat, it's fine, but don't list everything you ever thought might have happened. The whole wide world, just in case for like with the pro, like, it doesn't, doesn't work.

That's a, that's a great example. Now, Ella, as we reach the end of our chat, I'd definitely not be remiss not to talk about some of the data science use cases that you've worked on on Shop at Shopify. So what were some of the highest impact data science solutions that you've developed that you can publicly share? Of course. Wow.

There is a lot of cool ones. So I will tell you definitely we talked a lot about Shopify Capital, which is offering loans for merchants to like scale their businesses, which is amazing, it's validated, it's a data driven product and it definitely does have a massive impact on merchants and their life. We also have Shopify Balance. We do have our product classification as well as our audiences, what we call audiences, which is like enabling merchants to market better, which is the return on investment on ad spend for merchants so that they can actually scale, which is pretty, pretty darn cool. Because think about it, like ROAS tools or organizations that build. ROAS is actually usually either very data driven, so they already have large data teams or they use third party tools to help with that. This is part of the Shopify offering, which is pretty cool. Some of the ones that I'm personally very excited and invested in, like some of them are internal. So like our own forecasting family of algorithms. And like within the economical environment that's happening now, forecasting your forecasting merchant count or any of that is pretty hard problems. So this is pretty cool. The other ones is like Best Next Action, which is the recommendation engine I was telling you, which is when Shopify merchants start starting a business is not easy. There is a higher probability of failure because like entrepreneurship is hard. Not Shopify aims to make it as simple as possible and removing as many barriers as possible. And and because of that we do have this recommendation engine which is Best Next action, which helps you becomes your partner in your early journey to make sure that to get you to a successful start on Shopify as well on entrepreneurship in general. So there is a lot to be excited about and to be proud of.

I love these use cases and what I love the most about them is that of course there's a lot of value for Shopify that is generated from these use cases but it really also provides a lot of value value for would be entrepreneurs who would not have become entrepreneurs without these use cases potentially. And that's amazing to see. So connecting back then to the theme of the episode, kind of final question from my side. What were some of the biggest learnings for you from working from these projects?

Yeah, that's a great question. So reflecting on it, I would say number one is as I shared earlier, like always start simple because when you start simply create a baseline and you understand what's possible with the load lowest friction points. So even, even something like best next action instead of starting, we didn't start with like the fanciest algorithm that we currently have. We started with okay, what about we just organize these, organize this list like we're going to do analysis and like force organize them and then maybe we stack rank them automatically and then maybe we feed this machine learning in and then we iterate it on it. So starting simple made us understand the impact fact we experimented through so that we learned the value as we iterated making sure that we got check our hypothesis. So number one, start simple. Number two, experiment to learn and iterate to also not to fall into confirmation bias, right like to make sure that you're, you're really gut checking. Last but not least, creating a space for experimentation and mini sprints actually tends to surprise me every single time. Like I am, I'm a big advocate of it. A lot of our cool internal solutions started as a mini sprint that then stood up to become a fully productionized product after. So this was very helpful and I would definitely encourage for us to continue doing that and for others to use it.

That's really great. And maybe you know, on a personal note as well, what were some of the biggest learnings for you from going from an individual contributor to someone who manages data teams as well? Because that's a jump as well that's not talked a lot about in data science with its challenges and the different ballpark that you are in as a.

Data Leader, I'll tell you honestly, every day is a learning. But I'll tell you, back then, when I did this transition, I did it many years ago. But I think the hardest thing that, and I still see people who are moving from an individual contributor into a leader struggle with, is knowing to trust, to let go and create the space for others to learn and fall forward sometimes as an individual contributor, especially when you're at the top of your craft and this is why you got promoted into manager, you think it's like, oh, you just. Like, I can do it in 15 minutes. Yes, you can do. Maybe you can do it in 15 minutes and that other person might end up doing it in two hours, which is like eight times how much you do it. But, like, if you let them do it in eight hours today, tomorrow they will do it in two hours, which is like eight times the time. Like, if you let them do it into two hours, then tomorrow they will do it into one hour, and then the day after they will do it in half an hour. And then. You scaled yourself up as a manager, don't forget that your job is to work through others and lift them up around you. Because it's not like the best managers are not the smartest people at the table. The best managers are the ones who have very strong people around them, where everybody on the team lives, lifts each other up. So that's a key reminder. It's not just hiring the great people and getting out of their way. And I know this is a very popular quote from Steve Jobs, it's hiring great people and give them a space to learn and to level you up, and you level them up. So it's an environment of shared learning, and I always call it collaborative intelligence because you get together smarter.

That's such an awesome ending. Now, finally, Ella, do you have any final call to action before we wrap up today's episode?

So all I can say is maybe my final call of action is like, data science is a great field and there is a lot that we can do to still shape it. So have fun. Don't get stuck at a tool or a method or just, like, focus on the business problems. This is our superpowers. We are problem solvers. Data scientists are problem solvers. So focus on that, and I think a lot of good would come after.

Thank you so much, Alec, for coming on Data Framed.

Thank you. I was excited to be here and I'm happy to have the conversation. Thank you so much for having me.

You've been listening to Data Framed, a podcast by Data Camp. Keep connected with us by subscribing to the show in your favorite podcast player. Please give us a rating, leave a comment and share episodes you love. That helps us keep delivering insights into all things data. Thanks for listening. Until next time.