#292 Offline A/B Testing: Experimentation in Brick and Mortar with Philipp Paraguya, Chapter Lead Data Science at ALDI DX

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

In a brick and mortar retail chain like we are, this actually impacts your whole supply chain and thousands of people. Transparency is very vital. Keeping the stakeholders as close as I can from the beginning. This is why I want to have them close by so that they see the journey, that they see the steps and how we improve step by step, but also get a feeling of the effort that goes into this and the boundaries that we might have.

Hello everyone, this is Adel, data evangelist and educator at datacamp. And if you're new here, Data Framed is a podcast in which we explore how individuals and organizations can succeed with data and AI. As you saw in last week's episode with Vanessa Larco, it can be challenging to set up an A B test even effectively. While it is definitely challenging to set up an online A B test, like changing the design of a landing page or checkout experience, it's probably 10 times harder to set up an A B test in the physical world like in a brick and mortar shop. So I wanted to understand how that's done effectively and what are the challenges of building an experimentation culture in such a setting. Filip Paraguia is the chapter lead of Data Science at Aldi DX, AldiSud's digital transformation arm. If you're not aware of Aldi, it's one of Europe's biggest retailers. Throughout the episode we spoke about how to set up AB tests in the physical world, how to hedge against real world events corrupting results, how he spreads an experimentation culture at an organization like Aldi. Best practices in building data stories, the skills data teams need for the future, and a lot more. As always, if you enjoyed this episode, make sure to subscribe to the show, give it a review and thumbs up. And now let's get started. Hello Paraguya, it's great to have you on the show.

Thank you very much Adel. Happy to be here.

So you are chapter lead Data Science at Al D dx, which is Aldi Digital Experience, the umbrella department of all of Aldi's digital and IT department, Aldi Sud. Do you want to first explain what Aldousud is? For those who are not aware, Aldi.

Is an international retail company. We are in 11 countries worldwide and our discounted chain. So everyone who's gone to a supermarket basically can imagine what Aldi Suite is basically.

And we're definitely going to talk a lot about the applications of data science and of course, a B testing, given today's discussion in retail. So maybe to jump right in, you know, you've been working on a B test at LT Suite for a while now, maybe to set the stage, what makes a good A B test?

So I think a good a B test starts with a clear hypothesis, so basically a well defined goal and also a good understanding of what impact the test would have on your actual business. And I'm not necessarily talking after the test, which of course is a very important thing, but I'm also talking about during the test, which impact can doing the actual test have on your business in itself? And then I would say that a good A B test requires a good proper design. So to say, just to make sure that we have enough statistical power behind it to actually make sure that we can, well, reliably recommend something there. It's nothing worse than having a badly designed test and don't have any reliable results at the end.

And maybe I want to jump in as well because you talk about reliability and the statistical power of AI. A B test, I assume this is quite challenging in a brick and mortar setting. So maybe walk us through this particular challenges of running a B test in a brick and mortar setting for an organization like Aldi.

Sure. And maybe to give a short background for those who are not that familiar with a B testing to have in mind where we a B test often or where a B testing is coming from. And a B testing is most often used in marketing analytics settings, I would say, like optimizing campaigns, assessing impact of website changes. And these are of course very different from brick and mortar because you can very simply change a website, you can very simply change content and have it consumed by different types of people. In a brick and mortar retail chain like we are, this is difficult. Like, let's take a typical retail example of finding the right assortment. So what's the correct assortment to fit the demands of what our customers want? Well, you could think, okay, let's a B test this. Let's try out different products and let's see which ones have the highest revenue impact. For example. However, the thing is, once you start thinking about how to set this up in an actual setting, you come to, well, okay, if I want to do different products, I need to buy different products from different vendors, so I need different contracts. And then I need different distributions from different warehouses because of course I have actual product. And then I somehow need to have different assortment at different stores, which I wouldn't necessarily have otherwise, and maybe different promotions even. And so suddenly you're thinking about how this actually impacts your whole supply chain and thousands of people, while in theory you're just testing different products in different stores.

And one thing as well, because in a B test, you need to compare the A and the B within the test at the same time. Like, how do you do that in a store as well? Do you have separate spots in the store that are having different assortments, or do you compare it between stores within the same area? Like, walk me through kind of as well, like, how do you physically set up the test?

Yeah, exactly. That's the question. Right? So usually we would say maybe to take different groups of stores and test things there. However, you have to make sure that you can actually do this from a, for example, logistics point of view. And usually in an ideal a B test setting, you would even switch them around again after a period of time to have a more stable answer. But, but you can't really do this often in a real store because you would switch logistics chains too often. So it's very much finding a good balance between effort that you're putting into the A B test and getting reliable results.

And then maybe outside of this particular example that you just mentioned of assortments, what are other types of problems in the brick and mortar space that lend themselves to an a B testing approach?

The classical problems from a marketing analytics standpoint are things that are also interesting for us. And I'm not only talking about an E commerce or online world. Most supermarkets do have leaflets, for example, as marketing channels. And so promotion channels and seeing which promotion channels do actually work are things that we can a B test and that we can see if they have an impact. Same with testing out promotions or testing out assortment changes. So everything where we want to basically test for a particular impact is something where we can and we are doing AB tests.

So yeah, if next time you, anyone who's listening, you go into your supermarket and you see that there's a different leaflet from usual, there's probably a data scientist behind the scene deciding what's going on, maybe might be the case.

Yeah, yeah.

And then can you maybe share an example of a B test that actually did maybe had counterintuitive results or did have a kind of semi significant change from resolve from what you were expecting?

Sure. Although to be honest, I would have to go before I joined Aldi, but I guess it doesn't really matter. So a couple of years back I was the team lead for a marketing analytics firm. It was still a brick and mortar topic, but was not Aldi Stores. And back then we tried to optimize conversion rates for their E. Com area and we tried to change it based on customer target group based website design. And so we did segmentations of customers and basically created different content for the website based on which group they were part of. And we did this, we did the analysis and at some point we noticed, oh shoot, we don't have enough customer data at that point in time of our customer journey. Because one thing we forgot to take into account is well, at the point of people surfing our website, we don't have actual data of them yet because we get data after they bought, we get data after they did something. Right. So we had our or we tried to do a segmentation based on data we didn't have yet and that didn't really work. However, we didn't notice this right away and this was something that just didn't work out at the end.

And maybe coming back to the brick and mortar example, one thing I often think about is in a brick and mortar setting, in a physical setting, right. Even just going beyond brick and mortar, there's a lot more potential to have variables be corrupted by the outside world and in the environment. Like one thing that comes to mind, what if the weather in one area where you're measuring stores is much worse today than in another area and that's affecting how many visits you get. How do you create protections against this type of effect and dynamic when doing a B tests in the physical world?

Yeah, you're absolutely right. Because oftentimes in the physical world we have lots of variables in there that we cannot really control. Right. It's things like weather as you said. It might also be just location. Like in my data set, looking at two lines of store might be more or less the same and only a couple of attributes that are different. But when you then go to the physical locations it's very different because one might be at the mountains and one might be in the sea or whatever. Right. So it's very different than variables that we cannot really control. Foot traffic being another thing. And I think this Is oftentimes a thing of being aware of this happening, being aware of these things happening. Maybe getting data to try to have this in the data, like demographic data, geographic data, weather data being another thing. And to have them as an option in there, even though most of the times in my experience they don't really matter in most cases. There are some cases where it might be interesting. Like when I look at, I don't know, this is now a fictional example, but if I look at ice cream sales, weather for sure will have an impact. Right? But when I look at sales across our whole assortment, probably not so much because people go through the store if it rains or if it's dry.

Anyway, you mentioned kind of the challenges in the brick and mortar setting related to, you know, there's supply chain adjustments that need to be done, there's creating contracts with vendors. It's not just an ability to change the color of a button on a website. Right. As simple as that. So I feel like here you need to probably do 10x the amount of effort to align stakeholders, align department leads. Walk me through the challenges associated with alignment and creating kind of a common A B testing agenda and experimentation agenda. And how do you get people excited about doing experiments?

So I think, and this is a very agile mindset, I think the agile community came up with this sort of. And that is transparency is very vital. Keeping the stakeholders as close as I can from the beginning, I have the feeling that especially in the last couple of years with data science becoming more and more used nearly a commodity with lots of the things we're doing, it's still an education job with people outside of the field. It's still trying to make them understand how this works, how this maybe cannot work as well. Because I have the feeling there is a spectrum from don't trust this. This is the computer. Taking my job up to this is magic. This can do everything right. And there's like the spectrum. And you can find both extremes, maybe even in the same project. And so it's very much being transparent there, trying to educate. And this is very much which has become very important.

What's easier to deal with? Is it the folks that have really high hopes out of data science and AI, or is it the folks that have really high reservations when it comes to data science and AI?

Oh, that's a very good question. This might be also personal preference. I feel like it's easier the ones that have very high hopes because at least they come with a very high motivation. I don't have to sell them on the idea. I don't have to sell them on the idea. I just have to adjust what they are expecting while still keeping their motivation high. To motivate someone and educate them on something is for me at least more difficult because you need to educate someone who doesn't really want to learn potentially as well.

And yeah, that resonates with me as well. If someone has bought in already on the importance of doing this, even if their expectations need to be adjusted, that's much easier of a battle, maybe. What have you learned? Best practices when it comes to shifting people's perspectives, especially the naysayers, I'd love to kind of hear some best practice here because this is something almost every organization faces.

Yeah, true. And I think that resistance to these topics often stem from either lack of trust in a solution or maybe lack of understanding, because they don't really know yet. How can this work? How does this work? Or should this work at all? And so I feel like working with examples, simulations, maybe fictional examples, examples from other companies from the past might help there to gain trust in a solution. And also this is why I want to have them close by so that they see the journey, that they see the steps and how we get, how we improve step by step, but also get a feeling of the effort that goes into this and the boundaries that we might have.

Yeah, and maybe zeroing in on a B testing itself when it comes to this. Have you ever had a situation where you would encounter resistance even if the a B test had a definitive result, that we need to change something, for example? Is that part of the process as well, the education process when it comes to changing hearts and minds within the organization?

Yeah, I think so. Because people often go into an AB test with a notion or with an assumption of how this is going to go because people already have an opinion of what's the ideal solution. And of course, if the AB test then shows you that it does not actually agree with your hypothesis, you, well at first run into the wall there naturally. And so I think this is where having a good hold of or a good grasp of quality measures and so on gets very important. Because once you ran against this wall, they will try to find the things that are not working. The holes in your IB test, they're trying to see the hole and rip it open as far as they can. Because if there is one hole in it, why shouldn't there be another hundred ones of them? Right. And so this is why the analysis part, doing quality checks, doing quality control, good testing of my analysis is Very important. And I mean, honestly, of course you will do mistakes here at the end, right? And this will happen. And of course, this does not necessarily mean you're a bad data scientist. Everyone does mistakes. And this is where then code reviews, having other people double check and those kind of things are getting important as well.

Okay. And then, you know, like when the results of an a B test are accepted, everyone's aligned. Especially in these contexts, right? Where there's so many teams involved, there's so many business processes that need to change. How do you roll it out? What role does data science play in making sure that the outcome was dictated by the experiment actually is rolled out into the business?

It depends, of course, a bit of what sort of thing we're talking about, because we can a B test things that don't have a data science component. The testing itself is then the data science. But the thing we're rolling out might not be, but oftentimes we are a B testing, for example, software components that have data science parts in them. This is where, for example, then topics like machine learning, engineering and so on get very important. I like to think of, because people have different notions of what a data scientist is or what a machine learning engineer is, right? And you ask three people and you get four different definitions. And I like to think of this a bit of like during the development of a model or something. My data scientist is similar to what maybe a chemist would be in the chemistry lab and like creating the new material. And I'm in my lab and doing this. And now we're in a stage of we've tested this and now we want to roll this out, and then suddenly we have to become the chemical engineer. We have our example in our laboratory, and now we have to scale it out to an industrial setting, to an industrial plant. And this is where oftentimes, for example, machine learning engineers come in.

I think this marks a great segue to discuss how data science is actually done in an organization. Because in a lot of ways, when you look at universities today, I think a lot of universities teach data scientists to become chemists, right? But not necessarily. A lot of universities are focusing on how to teach data science to become chemical engineers. Often that's learned on the job. Or you become a software engineer who learns data science over time, right? So maybe first focusing in like outside of a B testing, right? What kind of problems do data scientists, LD or similar organizations simply focus on? Because there's often this hypothesis that it's only going to be chemistry. But maybe give me a Bit of the chemical engineering side, what does that look like?

I think and looking at its core, we're all about creating a business impact because at the end of the day, we're data scientists working in a business focusing on delivering some benefit to a customer while sustaining a healthy margin for us as an enterprise. And this is maybe not the most magical answer to, but this is the reality of what we as hired data scientists are doing.

Salaries come from revenue.

Exactly. Salaries come from revenue. And if I want the data scientists salary, I have to create revenue somehow. And so this of course can be science. But it is important to keep in mind I'm doing this to create a business impact and these benefits are what we have to keep in mind when creating solutions. When I am in university, this is not what I'm doing, this is not my mandates of what I'm doing there.

Yeah, that's completely true. And then maybe if you expand on the actual problems that you may encounter that require a lot more engineering skills or machine learning, engineering skills, as you mentioned, that you may not encounter in university, for example, as I said in.

Academia, my mandate is different. I want to generate knowledge, I want to answer questions and so on. So why bother with dirty data? Why bother with problems in my data set? Because these are harmful to what I want to do, what I learn. In reality, this is different, industry is different. Data comes from everywhere. Data might be the Excel file that someone in the business has created that has lots and lots of holes in them that I need to create some business of. And so data is messy, data is incomplete and we need to work on them, we need to engineer them to get to a point where I actually can build models, where I actually can well derive decisions from. And this is something that I work a lot with students coming freshly from university, junior positions, entry level positions, and oftentimes they're very surprised in how much work we do in engineering, how much work we do in data cleansing and how little in comparison we actually do.

The scientific part maybe focusing in on here, if you're to give advice to someone just going into the field right now, what are the skills that they need to develop the most to be able to succeed, succeed in a data role.

So people that come from university are often very fascinated in the modeling parts. Right. And I agree having some ones and zeros, predicting some complex context is very exciting and it feels a bit magical. But I mean, let's be honest, most problems that we have in the businesses are not using every state of the art method, are very easy are very simple. A simple regression might not only be enough, but might be the most feasible option. So I would say as an advice to impact more, on the one hand, data driven skills, engineering skills and so on. On the other hand, skills that are more soft skills, storytelling, problem framing and on those kind of things, understanding business problems.

Okay. And speaking of business problems and storytelling, I think communication. I often think about communication as probably one of the most important skills a data professional needs to have, especially if they want to grow in their career. Because you can be excellent technically, but if you don't have the communication skills that will put you as the most technical person on the team and then you stay in that role. Right. Because you're the technical problem solver. In your view, what are the core communication skills and soft skills like data professionals need to have to be able to succeed in a particular role?

And I 100% agree. I always feel like coming to a certain seniority. The difference in the ones that go further are more the soft skills than they are the hard technical skills. And I feel like knowing your audience and knowing what do they need at the moment to understand what I want doing or to give me what I need? Do they actually need to see what my print of feature importances for type of loss functions and my model representation as a couple of trees or do they need to understand what the business impact is of my analysis first? And do they need to understand what logical next steps come out of what I am actually proposing? And I feel like understanding these things of what your audience at that moment needs and tailoring then your message around these kind of things is a thing that can get you a step further.

Based on your experience and kind of in your own growth path as well. What are some best practices that you recommend for folks to become more better data storytellers, especially if they're presenting to non technical audiences over time?

I feel like you built yourself a toolbox of different things that you pull out during these kind of things. And for me I'm a fan of asking so why am I presenting right now and what do I hope to get out of this and what do my audience need or why are they listening to what am I saying? And that sort of focuses and frames me my presentation that helps me reflect on what do I now put into the presentation or what do I leave out of it? And this is, I just said feature importance is feature importance. Interesting. Also lots of the time for non technical people. Absolutely. But maybe not while I'm presenting results trying to get approval for something this fits into another conversation.

Okay, that's more likely maybe something that you share with a senior data scientist or a senior data leader looking to verify your results, but not something that you would share with a business stakeholder. Maybe. One thing that I see, especially with a lot of folks who come from technical backgrounds, is they join Data Science because they love to code. They're not necessarily people people. And then they often struggle, they hit a wall when it comes to improving their presentation skills. How can individuals grow their data storytelling skills? What are resources, tools, frameworks that help you break out of that shell?

I feel like practice is one big thing here. Actually to do this regularly and get feedback on what you're doing. I like to, in terms of very actual frameworks and so on. I'm a fan of using the Minto pyramid, like doing a top down example rather than a typical storyline, which is very counterintuitive for someone coming from academia at first, then learning how to build very good one pagers, summarizing what I actually want to do and visualizing data in a meaningful way. These are all things tools that you can put into your toolbox and pull out whenever you need them. But I feel like practicing and getting feedback on what you actually presented from someone in the audience is one of the key things that can help you get better.

Do you want to expand maybe a bit on that narrative storyline that you mentioned? Can you repeat what it was again and expand on it? Because I think that's something a lot of folks struggle with.

So there is this concept which is pyramids presenting or mental pyramid, which is you start with your key learnings, your key analysis, and you build for example, a slide or whatever containing everything that you want to convey in one slide. And then the next ones will be going a bit more into detail and then the next level would go more into detail. Why do this? Well, you start with what you want to convey and you can make actually sure that you are conveying it, because especially when you talk to upper management levels, it happens all the time that after you have an hour of time and they say I have to go in 10 minutes, what do I need to know? And then you have a slide deck of 30 slides and what you want them to know is on slide 25. And so suddenly you have a problem. And so when you do this pyramid style, you don't have a problem. They don't get every detail, but they get what you want them to get. And this is counterintuitive because coming from academia, for example, we learn that when we write papers and when we present, we need to start with our motivation, we need to start then go into definitions, we need to get into method, we need to get into how we did things. And then at some point we go into results and maybe risks.

100%.

And this is the narrative that all of us learn there when we go through these university settings. And this is because it's a very intuitive storytelling. But in management presentations or in business presentations, this might actually be problematic because someone asks you a question along the way and suddenly everything derails and you've not even said yet what you want to say, what you want to convey. And this is where this pyramid presentation style comes in handy, because you start with what you want to convey and then from there you can go as much into detail as you need to.

Yeah, that's wonderful. I love that. And I couldn't agree more. And then one thing someone asked me recently, I was delivering a training on better data storytelling and how to approach it is how do I know if my data storytelling is resonating with my audiences? How do you get feedback whether your presentation is working? One thing that you can see is who's engaged or not. But is there a more formal way as well to be able to understand whether your data story is sticking?

Sure. If you're in front of a smaller audience, you can see who's engaging and so on. Right. And to be honest, since everyone is more or less not so much in front of people anymore and oftentimes in front of the screen, this already gets harder. If people have the camera on, you might get a bit of an idea, but it's very different from standing in front of people. And then there is a point of when too many people, you also cannot really see what's happening anymore. And so I feel like this is where then the feedback comes actually in. So what I like to do is when I do presentations, have someone also in the audience, sort of a plant that I plant into the audience, more or less. Right. To either give me feedback after, how was it? Can I improve? Did I convey what I wanted to convey? You could also do this, of course, beforehand, practice rounds and so on, or just ask after or ask also at the end. Did you get what I wanted to do? It depends a bit, of course, on what audience you have. But more often than not, you can ask for feedback. People are most often keen on giving you feedback.

So that exactly resonates with what I was thinking around that. And maybe as we look into the future of bam, we're Talking about skills here, let's maybe switch gears, talk about the big elephant in the room, which is generative AI tools like ChatGPT and Copilot are definitely being integrated every day into coding workflows. Walk me through your experience with them. And then secondly, do you think coding will become less or more important for data scientists as time goes on?

I feel like you're absolutely right. In the last couple of months, years, however you want to put it, these assistants in coding get better and better and become more and more available. And I spent a good amount of time in the last couple of months being fascinated in how good coding support has actually become when I do my project, but also understanding a bit the boundaries where we're currently at. And that is often that while I do have to code less line by line by myself, doing the review and knowing how things should work are still very relevant. It's still me having to look over it to find the hallucinations and to find the things that are not working out the way it's supposed to. And I think this actually puts us in a very interesting spot because people always also ask, can we replace programmers at some point? And no, I don't think so. However, that being said, code that is generated is oftentimes, to be honest, as good as maybe entry level junior developers might produce in a faster time. However, then the question arises when we think long term, if we need seniors to actually do the review and make sure everything works out, how do we get them? If the getting experience part is not us, well, crashing our systems every now and then to get to a point of understanding what's needed. Right? And I feel this is a very interesting point which I don't see spoken about very often, that yes, code is getting good that we generate, but we need also the experience to have long term pipeline of people that can actually still do the quality assurance and can.

Ask the correct questions, what's the solution there? How do you imagine this? Because I imagine a lot of companies potentially in the future will opt for more coding support instead of hiring junior developers. How do you see the pipeline from junior to senior talent going, moving in the future?

I have two perspectives here. So one thing is this might be the data scientist in me. In reinforcement learning or in learning in general, there's this concept of exploitation versus exploration, trying things out and seeing what works. So exploring versus taking the best and quickest route all the time. And this dilemma you solve usually by exploring and exploiting at the same time. And when I transfer this, this could mean yes, you maybe have generative AI doing the coding part, but you also have juniors doing the coding part to get there. So this is then finding the correct balance for your company of how much do I want to invest in my future seniors that do the quality insurance and how much do I need the efficiency part, for example, this is one perspective. And the other one is this is not very different from, well, hiring consultants really, is it? So maybe the shift is more, less consultants, more generative consultants or whatever, while having our own internal people that are also junior, this might also be a shift that can be happening there. And so I think these are both.

Potential solutions as time goes on. How do you see the kind of the future skill set of data scientists that they need to develop what will make a great data scientist?

So I think understanding the code and understanding the foundations as will still be very relevant. You will need this all the time. And I think what's getting more important is this notion of asking the right questions which nowadays sometimes gets a bit well equalized with like prompt engineering, but I mean, understanding what's my goal, understanding how to get there, doing the task breakdown and understanding then the output that maybe my different generative functions give me.

As we close out our chat, maybe I'll go back to the beginning of it. Do you see certain categories of problems being more automated or streamlined? Do you see generative AI systems being able to run A B tests or experiments at scale? I had a chat with Ben Stancil once and he mentioned to me that one thing he's excited about with generative AI is not being able to just do A B tests with changing buttons or quantitative metrics, but being able also to run, for example, quantum qualitative surveys at scale on your website. And that kind of changes the nature of experimentation. So yeah. Do you see generative AI changing experimentation as well?

Yes, I think so and I agree with the point there. I think AI has always been automating things that we might find boring at some point or that are very repetitive. And with a B testing, do I see Genai planning how the best way for a brick and mortar store to run an A B test is in the foreseeable future, not quickly because of the complexities that we already talked about. But do I see an A B test result being automatically analyzed and getting recommendations out of it? Absolutely, and I don't think it will take too long because this is also the part which is very routine because I have the data, I run some tests on it and then I get some results. The more interesting part is the other part. And so that's why I feel like, yeah, this is where automation can come in.

It's going to be an interesting time. Maybe. Philip, as we close out, do you have any call to action to the audience before we wrap up today's episode?

We talked a lot about the A B testing and so on. And I think more of a mind experience, thought experience kind of thing maybe to the audience. Think about where in your life could you maybe use an A B testing just in your private life to get more data driven decisions. And you will find that there are a couple of things where you could a B test, I don't know, using one spice in your favorite dish versus another and see maybe data driven how the feedback is. And this will get you more into the mindset. Okay. A btest is not that complicated, but rather something I can also integrate into my private life.

We could all use a few experiments in our lives.

Exactly.

Cool. Philip, great to have you on the show. Really, really appreciate it. And yeah, see you on the next one.

Thank you very much.