#72 Building High Performing Data Teams with Syafri Bahar, VP of Data Science at Gojek

Hi, I'm Maggie, VP of Curriculum at datacamp and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

Hello, this is Adele Neme from datacamp and welcome to Data Framed, a podcast covering all things data and its impact on organizations across the world. One thing we always think about at datacamp, whether in webinars, white papers or podcast episodes, is the data maturity of organizations and what are the hallmarks of a data driven organization. It's oftentimes a combination of infrastructure skills, tools, organizational models and processes that enable data driven decision making at scale. This is why I'm so excited to have Chafri Bahar VP of Data Science at Gojek on today's show. Gojek is an Indonesian super app that provides more than 20 data enabled digital services such as food delivery, transportation, commerce, payments and more. It was the country's first unicorn and Decacorn for that matter, and Chauffrey oversees a large portfolio of data products and manages a variety of data roles. Throughout the episode, Chafrey discusses his background, the hallmarks of a high impact data team, how he measures the ROI on data activities, the skills needed in every successful data team, what is the best organizational model for data mature organizations, how COVID 19 affected Gojek's data teams, his thoughts on data literacy and data governance, future trends in data science and AI, and why data scientists should sharpen their math and machine learning skills in an age of increasing automation. Also, we'd absolutely love your feedback on how we can make Data Framed a better show for you and which guests you think we should bring on the show. I left a survey link in the episode description. Please make sure to fill it out as I greatly appreciate it. Jafre, it's great to have you on the show.

Yeah, sure. Thanks a lot. Actually, I'm happy to be in this podcast.

I'm excited to discuss with you the data science power in Gojek and all the best practices you've developed leading data science at such a data mature organization. Before we begin though, can you give us a brief introduction about your background And Go Jek's mission for those who may not be familiar with it.

Yeah, so my name is Shafshri. I've been in Gojekar for around three years. So prior to this I spent like most of my careers are actually doing modeling within financial institutions. So banks, asset management, insurance. Right. I'm like hoping from one type of risk into like another type of risk. And I think what's really nice about it because each of the risk type actually incorporates different mathematical models. So it also gave me quite some exposure to different quantitative techniques available out there. And then yeah, now I'm with Gojek, so I'm offering data as a whole actually. So I'm offering a portfolio related to consumer science and analytics at Gojek and basically oversee several job ladders. So we have, at Gojek we have data science, we have decision science and also business intelligence. So yeah, that's a little bit about me. I'm sure you're also curious about Gojek. Gojek is in itself, I would characterize it as super app actually like on demand app, like especially with the merger with Tokopedia, we're the largest in Indonesia for sure for now. And then with Gojek we have around 20 plus products, you name it actually. So we have ride hailing, we have food deliveries, we have logistics, even streaming service actually. So it's quite a lot. And then those are the different offerings that we can actually give to our customers in Indonesia. I think one out of four Indonesians actually has Gojek install, so that's a good thing. In terms of our drivers, we have around two and a half million drivers, so it's quite huge. Basically. I think together with our new friend Tokopedia, we've contributed to around 2% of Indonesian GDP. So it's pretty huge.

Yeah, that's massive. And I think for Western audiences the best similarity to Gojek Maybe would be WeChat if I'm not mistaken. So given the wide variety of tools and services Gojek provides, I'm sure data science plays a massive role throughout the value chain of different Gojek products and services. Do you mind giving a brief overview of some of these key areas of data science that deliver value for Gojek, whether for customers or internal usage?

Yeah, sure. So I think especially what I really like about Gojek, right, because data science really plays a central role in different product offerings that we have. Basically if we look at the whole life cycle of customers from acquisitions, activation, retention maximizations, right from their Booking values, even the customer services. We have a lot of like machine learning system actually powering those use cases. So it's actually quite broad but if I can name some of them. So definitely I think one of the first use case of Data Science at Gojek will be in our matchmaking engine. So basically this is an engine that's responsible to match drivers and orders, right? And what's really nice about it, I think from business perspective that it is a multi objective. So one can actually specify what kind of objectives that business wants to optimize and then the model and the system actually learn itself and then give the best allocation. Right? So that's number one, like a standard plain vanilla use cases are also you can see on search and the recommendation engine that we have. So we also apply that and I think especially like one of the biggest machine learning system as well that we have at Gojek, it's called Gobstopper. So Gobstopper in essence is a promo allocation engine, right? So it is basically responsible for like 80% of the demand generation budget of the company, which is very, which is really huge actually. So what this engine will do is that it will allocate the right vouchers to the right customers and we're basically combining counterfactual machine learning also with some optimization, abstract type of optimization in order to achieve that. So it's quite a combination of multiple things. So I'm pretty excited about it.

Yeah, that's a very large portfolio to manage and must be very exciting. I want to set the stage for today's conversation. So you're currently the VP of Data Science at Gojek. Gojek is quite a data mature organization. It was born with data in mind. And many organizations look at firms like Gojek as really the gold standard or North Star in where they want to become. So there are many levers towards succeeding in data transformation and cultivating high performance teams. High performance data teams is such an important part of this. I wanted to know what do you think are the hallmarks or characteristics of a high performing data team?

Yeah, I think for me like high performing team, right. There are a couple of characterizations that I would the way I would describe the high performing team, right? So number one, they need to be empowered, right? And I think the executive sponsorship plays a very big role in this because again if you look at the machine learning investment, right, sometimes it takes years, not even a month, it takes years until it manifests into something which can be measurable, right? So like having an Executive sponsor is very important. Definitely, number one. Right. And having that will ensure that the team will feel empowered. And I think to me empowering also means that there should be like relative freedom for the teams to try out different approaches. Right. And I think especially within gojek, we always encourage our data scientists to try out the latest techniques, the latest stacks. Right. And then they come back to us with the result of experiments. And this is also very interesting because this is also what we try to do also on company level. So when we talk to executives, executives expect already that we tried out all those different things. And then when we have the conversation, it's not about whether we can start investigating it or not. But hey, these are some of the new approaches that we've tried. Here are the results. Should we scale up? Yes or no? Right. And I think that also trickles down to the data scientists teams that we have. So empowerment is really important. Right. So ability to really be very agile with the approach and experiment, measure, do very fast iterations and then still related to empowerment is really also about, you know, have the courage to fail fast but then to learn as well out of it. Right. So that's a very important. And number two, I think everything needs to be measurable. I think we will discuss it also a bit later about measurability in all the different machine learning systems that we have. The first question is always, is the product engineering system that we want to integrate with, does it have sufficient capability to do measurement? We want to make sure that that's in place before we actually even engage in any machine learning system projects that we have. So that's also really important. And definitely number three, I think the team also needs to be empowered to do the decentralized decision making. For sure, it's again part of empowerment because being able to take decisions by themselves using of course, scientific methods will be able to again empower the team to make the right decisions without needing a very complex decision making structure.

Yeah, that does answer the question. And you mentioned here measurability of output. One thing I've seen you discuss and you've mentioned here is the importance of ensuring high leverage teams, improving the ROI of every data scientist on your team. And at the heart of that is the ability to measure impact. I think a lot of data leaders struggle with quantifying their work, especially in metrics business leaders care about. So can you describe what are the steps that you had to cross in order to reach such a high level of transparency on the quantitative value of your team's output?

Yeah. And I think definitely one very crucial ingredient of having a data driven conversation, right. Especially with the executives, it's really to, to start everything by asking the right questions. Like, like for example, what, what are the in or what is the impact? Where, where can I find the data and what are the measurements that we use as as north start? I think those conversations will actually trickle down also to the team level execution. Right? So that's definitely like ingredient number one, like having a sponsorship from the executive, right. And, and if you look at our CEO Kevin, so used to be head of BI actually for Zalora before Gojek, right. So he together with another senior data leader back then, Crystal actually like they built data organization, right. And so we understand really the values of having a proper data, asking the right questions. Having the, the right North Star metrics is also very important because using this metric actually we can march everyone towards the same direction. Right. So definitely that is a very crucial second thing. Like having a proper tooling in place is also very important. Having the proper infrastructure to do the measurements, right. Like I said, like having a mature A B testing capability for example is really important. Configuration management is really important. And all these different structures that infrastructures actually that we can think of, it is very important to have it in place, right? So that's number like layer number two and layer number three is really like being able to have the right methodology to measure as well. Because not everyone can be a B tested, right? That's also the reason why we have a dedicated job ladder for that. We call it decision science actually where they basically incorporate a lot of like a statistical techniques to really answer some of the most fake questions that we have within the company. Right. For example, how do we measure impact of loyalty? You can't really test loyalty, right. But and I think for, for that purposes we resort into many causal inference techniques in order to be able to do that. And then again there are so many choices. They really depend on the use cases that we or the questions that we try to answer. But I think what I want to say is that it is really important to have really proper scientific ways in order to ask the questions and to measure them and be really intentional about it. Like having someone who's really looking into this and really expert because it's a discipline on its own actually will really help a lot in order to get to that stage. Right.

So given that you've really emphasized creating this infrastructure and having this multi layered approach to measuring impact of data solution, how does that play into your decision Making process around which data solutions to further invest in down the line.

Yeah, yeah, it really influenced a lot of our decision making, right. Because again, machine learning investment or data science investment in general are quite expensive. So it is really important to kind of like being able to size the market size before we even start any machine learning engagement. Right. We need to be able to identify, I mean, if we compare five use cases, for example, are we talking about $1 versus $100 impact or are we talking about like 40, $60 impact? So I think it's really important to have that and we really use all these different infrastructure to really become the basis for your conversations. Different various conversations that we have within the company. For sure. That's how important it is to be able to have this measurement and be able to use that in order to make the right investment. And not only that, actually I'm talking, this is, I mean that's even the first layer. So the first layer is really about do you want to make investment, yes or no. Right. Second layer of questions can be. Okay, now if I want to tackle these problems, I have 10 different solutions, different data science, different ways to frame the problems. Right. I can frame it as unsupervised supervised reinforcement learning, for example. Each comes with its own degree of complexity and I think it is really important to be able to size that effort, you know, like the really to measure the trade off between effort and impact, putting in the nice quadrant. And then really, okay, this is the approach that we want to do, right? So that's how deep we went into decision making, basically using all those different infrastructure to make decisions on our data science projects.

You mentioned here the use of decision scientists. So from a skills perspective, given how prevalent and diverse data science is within gojek, what are the different roles you hire for and the different skill sets you think every data team should have?

Yeah, that's an interesting question actually. Right, so first of all, the one especially relevant for this podcast is data scientists. So I think for us, data scientists, data scientists at gojek, they specialize at building scalable machine learning systems. So inherently there are expectations for data scientists at gojek to be full stack and also to be able to apply good software engineering principles in building this machine learning system. So that's data scientists. And even just to make it even clearer, if I were to articulate. Right, so Data scientists at GoJek specialize at helping company making micro scale decisions very rapidly, like high, frequent micro decisions. Right. Okay, now it comes to decision scientists. So decision scientists, if I Want to contrast it with data scientists. Right Decision scientists will specialize in really helping making less frequent big decisions which inherently require a lot of statistical knowledge in order to be able to frame the problems and apply the right techniques, basically. And then we also have business intelligence, basically. So the role of business intelligence within GOJEK is really to make sure that we have one single version of truth where we look at things. They're also responsible in defining the right metrics, making sure the data is available. And they also to some sort of extent do the stewardship as well. Being able to map different business flows that we have and translate it into tables that we have just to make sure that we have a proper data models. Because I think that's also very important. Because if we cannot really do a lot of like the advanced analysis that, that, that we may want to do if you don't have a very solid foundations on the data. Right. Having single version of three is really important. Having reliable data that's that that does not break once every week is all very important. So yeah, so those are different Personas. And of course you also have ML ops machine learning engineers as well that we hire within gojek.

I'm excited to cover this more as we're talking about data teams. You know, oftentimes organizations undergoing data or digital transformation struggle with identifying the best possible way to organize their data talent. Some organizations gravitate towards like a centralized center of excellence model, other towards an embedded model where data scientists are integrated within functional teams. How is the data talent organized at gojek?

Yeah, I think for us it's also. It was also an evolution, I would say. We've actually tried various different models. We started with metric organization. I think that was good for a very small team because we want to make sure that everyone follows the same practice. So there needs to be a centralized voice in terms of how we should do things. Basically what we do is that what we did back then is that we have a central team, we dispatch them to different products. Right. But we're basically we were acting as consultants pretty much back then. Right. But then as the, as the team grows, as the use cases grow within gojek, that model was not sustainable anymore. So what we do now, we are operating according to the federal system, I would say. So each of the head of data now we think Gojek or data leaders, they have their own domain that they need to take care of. We are very embedded within the within business team and also product teams just to make sure that we really like feel the heat as well. We are really invested. We have skin in the game. And it's also really like allows us to think bigger than the problem statement that we try to solve. That's also very important. Right. And just to make sure that we have consistencies in terms of our practices and consist of our career path and et cetera, we form a council basically within data leaders at googek, just to make sure that there's just. We follow the same practices and standards as well. So currently we are at the phase where we are fully embedded within business team and product teams.

So do you think as an organization increases in data maturity, the operating model needs to evolve with the organization's data capabilities and it moves to something that is more hybrid?

Yeah, I would say, even though that's inevitable, I would say because otherwise the organization becomes too complex to manage. Right. And then for the functional leaders, the leader in the central function, it is also very difficult, especially if we look at the domain of data science. Right. It needs like, if you wanted to have like meaningful conversations like a leader, a data leader needs to have two, three layers, deeper context, depth, basically in order to be able to have a meaningful conversation with the team and with their business leaders. And then if we putting it that way, it's just very difficult to be able to maintain knowledge like having a context, especially within various domains that we have at gojek. Right. So I think to me then it's inevitable to go to that model.

As you said, skin in the game is really important because otherwise data scientists are just creating analysis on an ad hoc or support basis and that doesn't necessarily maximize the impact.

Correct. And I think one thing also that I try to encourage, right. Is that I try to encourage that traditional data people in general, data professionals in general are seen as service provider. But what I try to set as well within the organization is that data people are not service provider. We are thought partners. Right. Meaning that we need to be involved at the very beginning when the problems are being formed, because that way we can actually give a good recommendation, we have a skin in the game, and so on and so forth. I think there are so many benefits of doing that.

Do you think then that data scientists often end up not really forging a sense of business acumen or understanding of the use case they're working on? And do you think data teams should focus on instilling that business acumen within their data teams?

Yeah, I think so as well. I think like giving a big picture to data team is really important, right. To be Able for us also to think beyond the predefined problem statements that are given to us. So yeah, I think it's really important to have that right. And I think I'm not saying that all organizations work that way. I think it really depends on the type of problems that we solve. I think for a very hyper optimization type of problems it makes sense, right? Then one will just basically stick to that problems. We really go 100 layers deeper into the problem and then really solve it. But especially for problems with quite with large degree of ambiguity, I think it is really important to kind of like being involved in the conversation, right? Because it can go to multiple directions, especially in the objectives that we want to solve. So I think that's how I would differentiate between when especially like a data team needs to be involved at the very beginning or it can be that the problem statement is very clear, right? We want to optimize conversion, for example by building Recommendation Team and then Recommendation Engine and then it is very clear that we need to hyper focus into that problem.

So, you know, as a data leader and someone who manages multiple data teams that work on a portfolio of different products and services, I'm sure the COVID 19 pandemic greatly affected your team and the different data science solutions you work on and maintain. Do you mind walking us through how you dealt with concept drift affecting your models and more importantly how you managed to sustain high performing data scientists despite massive uncertainty and stress.

So I'll basically address it from two perspectives. Number one is that how it will affect how Pandemic has affected our team. So I think in terms of the effect of Pandemic itself, or even more specific, the fact that we need to work remotely, it didn't affect our team that much because even before Pandemic we are a distributed team already. So our teams are actually distributed in several cities, right? We have teams in Singapore, in Thailand, in Vietnam, in Bangalore for example. Even some of our colleagues even work also like from outside Asia. So it was less of an issue for us. One thing that we doubled down was that we need to be very good with documentation. Everybody needs to be very good at documentation side because the fact that everyone needs to work remotely, meaning that the one powerful means of communications will be via documentation. So we take documentation seriously for sure. We update always a lot of confluence just to make sure that the model has proper documentations, it can be followed. There are links to data source, to GitHub and multiple other things. So that's number one. Number two, in terms of the effect to the model. And I think even before Pandemic, we acknowledge and we realize that we work on non stationary environment already. If you look at the market, Right. And especially the fact that we dominate some of the markets, meaning that any changes that we do to our model can potentially change the behavior of customers. Meaning that there are also like a lot of feedback loop. Right. Meaning that the market will change anyhow and then we need to take it as business as usual, not as a, not as a phenomena. Right. But it's just that we always need to make sure that there are no drift in our features. For example, we need to make sure that we have a frequent training. Right. And more and more we realize that we need to adapt like a bit more adaptive learning techniques in our modeling in order to be able to cater the changing market situations, basically. Right. So I think for us maybe just to kind of like a summarize, it has been business as usual, but of course in terms of the business itself, it has largely impacted by pandemic. Right. So that is for sure we saw that. But in terms of the way how we work and how we basically create and monitor our models, nothing changed significantly.

Yeah. And I think this is where the data maturity of GOJEK comes into play. I think a lot of organizations realize this year that they really need to invest in mlops and the ability to monitor and update models in production. And that may not have been a problem at gojek. Because that's a capacity you guys already had.

Yeah, exactly. And I have to give kudos also to the data Science platform team. I think they've done good jobs at providing us with all the different infrastructure that we need in order to monitor the model performance in real time, having a feedback loop, deployment technologies and then so on and so forth. So they have been amazing.

That's incredible. And I'd love to expand our conversation beyond creating high impact teams. You know, as we discussed earlier in the episode, GOJEK is a really highly data mature organization that lives and breathes data. What do you think outside of high impact teams are the characteristics of a data mature organization?

I think it really lies not only on the tangible things that we can see and touch. Right. But also I think in the spirit as well, it's very important and I think one characterization, but. And this is a bit less also about Data team, but it's just an organization as a whole. Right. We need to live, breathe and then we need to use the vocabularies in our day to day conversation. Right. We need to really like Asking about correlation versus causations bias, P values, you know, like what are the Bayesian drift? I think that should be a part of the day to day comprehension in order for an organization to be labeled as a data mature organization. So it's really like inherently in the culture. I think it's a bit more meta than just tooling and dashboarding. I think that's really one of the characterizations and I think it's really important as well, I can't emphasize it more, is for the leadership to really give, to set an example. Because everything starts with asking the right questions to product team, to data team. Right? Because that questions actually will trickle down a lot of things. And I could maybe give some examples in the. Like back in the early days when we say, hey, we want to measure, for example, what was the effect of having a certain loyalty membership, right. It got us thinking like hey, we haven't had any infrastructure yet in order to measure these things, so we need to develop something, something, something more. Right. And then we started to explore and we saw maybe we could approach using instrumental variables, randomized encouragement, for example. Right. And then that really triggered us to build capacity capabilities around that as well. Like hey, what are some of the instrumental variables that we can generate as a company in order to help us measuring the marginal impact of a certain phenomenon which cannot work, which cannot be handled using a traditional AB testing. That was just again me trying to illustrate how important it is to ask the right questions, especially for data mature organization, because that will refill all the flows that we need to build from data perspective.

Couldn't agree more on the importance on the spirit and culture when it comes to creating a data driven organization. I think really that's the main differentiator you mentioned here, infrastructure. And one thing that's quite impressive about Go check is how geared the infrastructure is to towards creating high impact data science. I've seen you speak about this in other interviews and panels and this is something heavily featured in the GOJEK Medium blog. Do you mind walking us through the different technologies and infrastructure level in Vinovit? Do you mind walking us through the different technologies and infrastructure level innovations gojek has undertaken in order to facilitate high impact data science?

Yeah, so I think there are a couple of. Right. So we can talk about it from MLOps perspective. We can also talk about it from like really like the, you know, like the downstream data engineering stacks that we have. Right. So I think one thing that I really like about gojek is we like to develop our own solutions Actually especially when we think that the third party solutions do not cut our, you know, cannot actually cater our needs, right? So we've actually developed a lot of like in house system as well which were essentially also like a wrapper or of some of the recent technologies. And for example we talk about like a storage, right, for example. So we built a lot of like a wrapper around that. Just to give an example, we have built around 20 to 30 data engineering tools within Gojek. In order to help us moving data from one place to another place, we want to get our different transformations, for example data cataloging. We also have a solution for that. We've also built Optimus for example which is a CLI for doing data transformation. So we built quite a lot of tools, right? And that's only from data engineering perspective. So 2030 tools. And then when we look at MLOps or MLOps Steam data science platform, I've actually built quite a lot of tools for us. I can maybe quote, I can maybe like tell like two or three of them, right? For example Merlin, right? So Merlin is a tool that has been used by data scientists to deploy models. So what used to take two or three weeks to deploy model, now it takes like 10 minutes. For us what we need to do is just like saving the pickle file, the binary file, right? And then so we combine like we stitch different technologies like ML flows, Kubernetes deployment, Docker, multiple stuff actually like putting them into one, creating a simple abstraction for data science. And then all the things will actually just be managed by our data science platform team, right? So with Merlin for example a data scientist just to need to save the binary file, be that from scikit, learn Pytorch for example tensorflow put it and then there is like certain API link that we need to call it will automatically upload it, deploy it in Kubernetes cluster it will create logging system monitoring system and drift monitoring as well. Like all in one single go, right? It's pretty nice. We also have for example FIS is another important a product that we have we co develop with Google, right Which are also being used by several companies now. So this basically allows us to decouple the feature surfing and also like the model surfing basically by just doing, by basically just creating one layer for training for example we can use the same abstractions also to serve the model, right? And it also like provides good discoverability feature registration, for example an ability for example to do historical surfing or historical batch calculation or like an online real time surfing, right? Just using one Layer or one generic abstraction. Right. So it's pretty cool. It's quite a lot of stuff that we've done. We also have Turing of course and many other things like Clockwork is also another product. But I'll just suggest the audience just check our blogs and see what are the different tools that we've built.

Yeah, I highly encourage everyone to check out the gojek blog which is truly a showcase of amazing proprietary technology and tools developed by the team. Are these solutions open source Jafrey?

Yeah, I think most of them by now. So I think especially our data engineering tools, I think it was just recently like one or two months ago I think we started to open source the tools that we've built in host.

Yeah, obviously none of the above we've talked about so far is possible without high levels of data cleanliness and quality and organization wide data governance. Do you mind describing how important data governance is for scaling data maturity?

Yeah, for sure. And I think that data governance is often overlooked actually. Right. It is often taken as granted while for us it is really important because especially like if we look at the like in Indonesian like regulatory framework we are very basically like a strict and we really also want to keep the trust from the consumers that we have. So we are actually very serious about like having data governance even we have our own data protection officer basically. Right. We have a council of data governance that basically that who decides in terms of who should get access to which column, what type of NDA we need to like sign right before we can get access access to that. Recently we have launched also you also have launched a tool called ocean or data warehouse, new data warehouse tool basically like with oceans we can basically separate entity. We can really like govern. We, we can separate between presentation layer and data mart layer for example and give like a very specific access to that one. So I think for us what I want to say basically it is very important things for us. We do have a council, we do have a governance process in terms of how we make use of the data that we have. Right. And of course like various tools that we also develop in house in order to help that. Right. And I think the way we work it's always first we discuss what are the things that we need in terms of having a proper data governance and data protections and then we kind of like build the tools based on that requirement.

And where do you think the data leadership's role stands when enabling high quality data across the organization? Can you briefly describe some of the features of Gojek's data Governance program.

It is very important actually, right, Because I think there are a couple of roles that the data leader plays in this whole scaling of data governance. I think number one is really in terms of setting expectation. I think it is very important, right, because. Because without having a clear expectations, again, this is like an area which is sometimes being overlooked by people. So setting expectations is one thing. Second is also about education, having a proper education, why we need it, right? Why for example, a company will need proper data governance before they go public. So I think those are the second things, right? And I think the third thing as well, there needs to be a bridge, right, between like various stakeholders basically in the data governance, right? Because we need to bridge multiple stakeholders, right? From regulatory perspective when consumer. Consumers are also our stakeholders basically in this. And also like different functions, different products also within gojek. So I think bridging is also very important, right? So, so three things really, like, like set expectation, bridging, right? And the. And the third one is also education.

In terms of tooling, what are some of the tools that you use for data governance at gojek?

A lot of these things that we basically develop, we develop these capabilities in house. We create a lot of wrappers around it and also creating, for example, tools to really be able to show the lineage of the data and also to understand, for example, whether this comes from reliable tables that we maintain also on regular basis. Because historically from the hypergrowth legacy that we have, we had a problem like data that grew organically, right? Like really, like it's all over the place. We created the right table from derived table, for example. So it ended up to create like a forest of tables. So we are also in the process of cleaning that up as well. So. Yeah, but I think in terms of tooling, a lot of these capabilities we tried to develop in house, right? We might use a third party tool, but I'm not very informed about that, to be honest.

So I'd like to pivot to discuss maybe organizational data literacy. You know, as a data science executive, your role consists a lot of gaining executive buy in justifying resource allocation and all of that fun stuff. What do you think is the proper level of data literacy executives need to have in order to be one productive in these conversations and to critically assess the success of data projects, initiatives, right?

But I think this is something which, which we can't really control. I think in general, if you look at the companies, right, like companies, especially when we look at the different spectrum of companies, there will be like a different type of leadership Leadership also like within those companies will come from different background. Right? So it's kind of like the things that we can't really control. Right. But I think it's really up to the data leadership to be able to articulate the message really well to them as such that. Well, there are some basics at least in data, of course, but just assuming that the basic is there, it's really up to the data leaders in order to cater to their audience in terms of how they want to basically try to push some certain agendas with respect to data. Just as an example, for example, if I want to push more for resources for data science, what I will do, as long as the counterparts are rational, we can basically have a fact based conversation. I will bring my data and say hey, these are like three people team we've generated or we save this amount of money per user per data science. Actually we can actually normalize it to the gains per data science and then I can actually use that as an argument to be able to push for more resources. So that's just within the context of data science. But I think with the context of project like data governance for example, I think especially with respect to data governance this is just something that we need to do. I think it's the hygiene stuff, right. So there is just really no other ways to not doing that, to be honest. So I think in order to have a productive conversations on this, I think definitely the counterpart needs to have some certain appreciation for data. Right. And I think especially for leadership who were basically conceived in the last centuries, I would say I think the literacy around data should be fine. I think should be okay.

That's great. And you've mentioned multiple times throughout our chat maybe data culture, data spirit and how important that is. I'm sure it also relates to creating very strong self service analytics capabilities within an organization. Where do you think the data team's role is in creating and forging this culture throughout the broader organization and the use of self service analytics? And what are some of the best practices you've adopted at Gojek to sustain this? And do you have any lessons to share there?

Yeah, that's a very good question actually. So there are a couple of things that we are trying to do with respect to self service analytics. Right. So number one is really to have a proper and reliable data. That's really number one. So the first things that we did is actually we tried to fix the basics first. You'll probably remember all these different problems with thousands of tables that we that grew organically in the past, right. We tried to tidy that up first, right. Just to make sure that first of all it is reliable. It was based on like a reliable source of data. Right. And then also like being able to create a proper data mart on the top of that. So that's definitely like the basics that we need to take care of then when we go to the information retrieval. So there are a couple of things that we are trying to do. Basically number one is also encouraging our analysts or business intelligence folks to be able to create more dynamic dashboard. So we use tools like Streamlit for example, to be able just to create a complex visualizations, to be able to cater as such that business people, if they want to create a story, for example, it's just based on like several clicks and intuitive enough, right? And then for example buttons will appear as soon as they are interested in some certain domain and just to help them navigate with all the various informations that we have at gojek. So that's another things that we try to do and another thing is that we try to develop also like another in house tools basically that, that maybe I can't really be specific about the name at this moment and what it does, but pretty much the capabilities will be like we need just to define dimensions and measurements and then filters as well. So only three things that business need to know. So they can just like drag and drop things, they can just add dimensions, they can just like add measurements. And then those like inside that the needs can just be produced nicely. Right, but it also requires, I'm not saying that we're 100% there, but because it requires a lot of standardization and especially given the fact that we have around 20 plus products, it's not necessarily the easiest job to do to be able to standardize everything to achieve that state, but those are some of the things that we're thinking and are actively doing at this moment.

And on the consumer end, where does data education fit within the broader organization? How important is data literacy Education is within ngojek?

Yeah, that's an interesting one. Right. So because we also like realize a lot of our consumers come from various backgrounds, right? So we are actually also actively pushing for data literacy. I can maybe mention one program, right? We have a program called Data Hero. So this is like a program which aims to educate consumer data consumers, basically teaching them SQL, teaching them the basics of data. Right? What are like a data warehouse, data mart, data lake and things like that in order basically to help them help themselves, right? Exactly, exactly. That is really crucial, I think in terms of like creating this awareness. Right. And of course it also helps a lot because most if not all of our leaders are very like a data. They have like high data literacy. Right. So they're always encouraged by the top leadership to keep asking about data to their people. Right. As such that people realize how important to have a data driven conversation and it creates the urgency for themselves also to educate themselves around how to use data.

Right. That's great. And I want to cap off our conversation by discussing some trends and insights and where you view the role of data science and AI playing in the future. So Goja creates so much impact throughout Indonesia and beyond with its technologies. How do you view advances in AI and data science further fueling value for Goja customers? And which advances are you excited about the most?

Yeah, I think there are a couple of things, right. So I'm most excited about the use of like especially the, the rise of causal machine learning actually. Right. Because a lot of things that we do inherently. For example, if you look at particular domain on prom optimization engine, right. It's all about. It is very natural to frame it as causal problems actually. Because just to give you an example, right. For example, if you want to do churn prevention, it will not be very useful to predict churn because it will create vicious cycle. Right. You predict people who are inherently very difficult to resurrect anyhow. So what we need to be able to do is to be able to be able to predict when they will be churned and then also to understand which treatments work most effectively to prevent them from being churned. Right. So to be able to frame it that way. And I think I'm actually very happy especially like in the recent years we were able to frame causal inference into machine learning as such that we can actually take advantage of the, like how good machine learning in dealing with sparse data like high dimensional data. Right. That is very crucial. Back in the old days we needed to manually specify confounders, for example, but now with the recent advancement in causal machine learning, for example, we just like put the data over there and then being able to gain the, you know, the marginal impact estimation, for example, and the data will, the algorithm will basically like learn, you know, like which, which are the most probable confounders. That's if you use, if you use confounders for example, in techniques. Right. So that's, that's number one, I think number two also I see that there are, there's quite a promising future for reinforcement learning type of algorithm. I'm Actually very excited about it. Right. And if you kind of like go down a little bit also like under the, under the hood. Right. Especially on the Markov decision process, I think it has a very promising application as well. Especially within the context of dynamic market. Right. Because we want to be able to have algorithm that can learn online basically without us needing to download it first, training offline, pushing it again to production. So as much as possible. Exactly. As much as possible want to push into that state where it's just basically like a learning by themselves. And if you look at especially within the domain of behavioral modeling, especially within the context of marketing and prompt optimization engine, there's quite a lot of areas where we can at least explore the potential applications of that. So that I'm also very excited about. Of course the recent advancement also in GPT3 you mentioned about AutoML. Right. I think that's also a very exciting development as well within the field. Right. And especially with GPT3, I think we just barely sketch of what can be done with it. So yeah, I think I see a world where data science solutions will be commoditized. There will be a lot of like out of the box solutions as such that as data scientists or data professionals what cannot be replaced would be really the creative parts of it. Right. So I would really, I would highly encourage especially for aspired data scientists out there to really like sharpen the problem solving skills like creativity being able to use these different tools. Because again, you know, like we might only estimate models with one single click. Right. But still it requires in depth understanding of the mathematics and statistics in order to be able to interpret those models, those solutions. Right. And being able to take decisions out of that.

So yeah, I know you're someone who's a big math fan and I've seen you argue for why data professionals should develop a deeper technical understanding of the models they're working with. How do you reconcile this worldview with increasing automation technologies for data scientists?

So I think one thing and I think it has a good purpose by the way, like all the different automations which are being created, it has a purpose basically to democratize machine learning and AI. I think for some certain problem areas that makes sense because it will basically especially on the repetitive stuff. Right. And I think I remember also what Andrew Ng quoted. So everything that can be done within one second is actually like a prime use case for machine learning or AI. But I think there are also more and more we discover also a lot of different type of problems which cannot really be solved within one Second, these are very ambiguous problems about for example distributing vouchers, how to develop users into more mature state, for example. So it's a very tricky one. Right. So again what I want to say is that it has a good purpose but it might not able to tackle all the industry problems that we face currently. Right. Especially on various domains that can generate like a lot of impact. So that's number one. Number two, I think even though there will be quite, there will be a large degree of automations in the future, but I think it is also very important to understand the mechanics underneath these, make these automated solutions actually for humans to be able to make wise decisions on how to use it. Right. Being able to interpret the byproduct of those estimations, for example are also very important, you know, like how to instead of like applying it blindly. But again there are like areas where we can comfortably do that. But there are also like another areas that you can't just apply some mechanics blindly. It is really important to understand the mechanics and to really understand what it does.

That's awesome. Finally Jafri, any call to action before we wrap up today?

Yeah, I think there are a lot of and I just want to also relate it a little bit with this advancement in automations of machine learning, right. And I think some people might think that hey, our jobs will be replaced, it will not be sexy anymore, right? But I think contrary, I would say more and more different breed of, of data scientists actually will be needed in the future, right? People who are able to solve problem applying first principles and try to combine the various available solutions, stitching them and being able to decide which solutions actually can solve a particular problem. So that's like a word of encouragement basically to still invest yourself in the domain. Especially in some emerging countries like Indonesia, we barely scratch of what we can do in terms of impact, right? Especially if you look at the structural inefficiencies in some of these countries, there's like huge opportunities for data professionals really to create impact over there.

Yeah. Thank you so much Shafi for the insights. I really appreciate it.

Yeah, you're welcome.

That's it for today's episode of Data Framed. Thanks for being with us. I really enjoyed Shafry's insights on the data science powering Gojek. If you enjoyed this podcast past, make sure to leave a review on itunes. Our next episode will be with Shamik Kundu, former group CDO at Standard Chartered and current Chief Strategy Officer at Truera. I hope it'll be useful for you and we'll catch you. Next time on Data Framed.