#209 Effective Data Engineering with Liya Aizenberg, Director of Data Engineering at Away

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

It'S important to create a vibe and atmosphere where everyone in the data team feels like they can come up with a cool new ideas and try them out. But at the same time, we need to make sure the team knows that it's important to focus on ideas that actually help the business. It's like, hey, let's try new stuff and see what works, but let's also make sure it's useful and get the job done. It goes back to Result Driven. You always want to bring the results. You don't want to just spin your wheels for no reason.

Hello everyone, I'm Adele, data evangelist and educator at datacamp. And if you're new here, Data Framed is a weekly podcast in which we explore how individuals and organizations can succeed with data and AI. In a lot of ways, the unsung heroes of the data functions are the data engineers. Data engineers act as a force multiplier, enabling everyone who needs data to have proper data access. So how do you build a high performing data engineering function? How do you make sure that data engineering teams are closely aligned with business value? Enter Leah Eisenberg. Leah is Director of data engineering at await.com, she has more than 20 years of experience managing data engineering teams and shared a lot of her wisdom on today's discussion. Throughout the episode we spoke about what makes a data engineering team effective, how to be agile as a data engineering team, how to best approach aligning the data engineering team's work with business value and a lot more. Also, just a note, the audio on this one is not the quality you're generally used to on Data Frame, but I hope you're still able to get value from it. We'll do a better job next time. If you enjoyed this episode and the Dataframe podcast, make sure to rate it wherever you get your podcasts. And now on today's episode.

Leah Eisenberg it's great to have you on the show.

Glad to be here.

Anil so you are the director of data engineering@waitravel.com and have been managing data engineering teams for a while now. So maybe to set the stage for our conversation. What makes an effective data engineering team?

There are few things I highlighted for myself that I find important to be data engineering successful is trust, collaborative work, not afraid to make mistakes, transparency and knowledge exchange opportunities. People should have an opportunity to exchange the knowledge within the team. I can go deep dive into each and every thing I identified, so we'll.

Definitely deep dive into those. What we want to focus on first maybe is building a data engineering team from scratch, right? If we take a step back, you know you mentioned building the right team, knowledge exchange, building trust, collaboration. If you're building a data team from scratch, there's tons of data leaders listening here on the episode trying to build their own data engineering teams as well. Let's focus maybe first on the type of roles that you would hire. What type of profiles do you look for in an early data engineering team or as an early hire in a data engineering team?

I think I identified the important traits for myself. I'm looking for a spark, a spark and passion for the data and passion to learn new things. Good understanding and knowledge of data engineering principles are very important. And good personality, open mindedness, eager to learn and help others. I find this important and passion, passion about data and empower your teammates.

And then you mentioned here kind of the personality if you deep dive into it a bit. Nor what are the kind of the cultural traits that you look for in an early data engineering hire.

I'm looking for the friendly and open minded people. I value people who take ownership of their work, who always finish what they start. You know, sometimes people get sidetracked not finishing things, starting one thing, then sidetracked to do something else. So I value the people who finish what they started, not easily distracted. Also good communicators, people who speak out, it's very important to speak out. And also the people who are not afraid to change because our industry evolves all the time, there are a lot of changes. Even within the one company can be a lot of changes. And people who are not afraid and easily adapt to the change are very important. And also result driven folks are very important. It's amazing trait. So you want to see the result. You want to make sure that whatever you do, you produce the result and outcomes for organization.

Yeah, that's really great. I couldn't agree more on results especially, you know, we're going to talk about how to focus on the Right. Projects that matter because that's a big trap. Data teams can fall in and maybe you know, we talked about the cultural traits but in early data engineering hire, I'm sure you know the technical skill set is so wide, the data engineering ecosystem is so fragmented. There's so many tools, so many skills to adopt here. What are kind of the technical skills you also look for in an early data engineering team?

It's very important. Like today I'm looking for the foundation of the. Every data engineer should have a good knowledge of a Python, SQL and relational and non relational databases. This is foundational all in knowledge of relation non relational database as well as Python. Also there are a lot of hide items today on the market. It's very good to have a knowledge of virtualization tools such as looker and tableau because you build all your data marts, you build all your data, but you actually need to serve this data to the business. So visualization tools are important. That's the tools you're serving your data with. Like name a few Looker, tableau. There are very good integration tools that are available today. Stage 5 Chan Matildian are currently in demand. Analogy of DBT is important. This is the tool that you build your SQL models. But I want to highlight SQL in Python.

I mean SQL is the lingua franca of data whether you're a data engineer or a data scientist or a data analyst. And when you mentioned kind of these canonical tools that you mentioned kind of DBT mation, SQL skills, what stands out for us today is kind of like the must have tool knowledge, especially in a modern data engineering team that you need to have, you know, outside of traditional tools like here I see a traditional quote unquote between Python and SQL.

I think any knowledge of IPAAS solution, it's integration platform as a service, it's a good knowledge of integration tools, knowledge of airflow, knowledge of any cloud, aws, gcp, Azure, any of this, knowledge of one of these clouds or multiple clouds. It's very important to today and you.

Know, as the team grows and becomes more complex, at what point do you decide when it's time to bring in specialists? Like what kind of roles do you start looking for? What are those specialist skills looking like? And yeah, walk me through that.

First you always want to see if you have a potential specialist already within your team or if you can rise one up. You want to empower the growth of your immediate team first. As a leader I have a periodical check ins with my team members so I have a good understanding and good knowledge, what is people want to do, how they want to move forward in their career. But there are cases when you project is required very specific expertise and you don't have it within your team and you have no time to train your folks. In this situation I usually source the expertise outside of the company to make sure the project is moved forward. We're not blocking anyone so everything goes as planned. However, it's important to pair up this expert external expert together with your team members so knowledge stays internally and your folks are learning something new. So it's very important to pair them up so they can work together. Your team members can learn new things while working along the side of this external SME.

Yeah, that's really great. And you mentioned here bringing up people to become specialists and that, growing people and training them, what does that look like in practice? I'd love to know how you've approached upskilling data engineering teams so that they can specialize into, you know, their respective.

Specialist roles as you grow your data engineering team. Once you have your periodical check ins, you have understanding. Some people would like to be technical managers, some people would like to be people managers. Not everyone want to be people manager, not everyone want to be a technical manager. So you work with your folks to see what's interest of theirs and you also identify the strengths of your team members and based on the strengths you assign them to the project, give them the work to do and based on their interest. It's very important to grow your team and keep their interest and keep them excited of the work they do.

And we've been really focused on the talent side of building a data engineering team. But I think a big question that a lot of data engineering teams, especially new ones, have to face is what is the tech stack that we want to invest in. We mentioned like between aws, Google Cloud, Azure, right. As a cloud service there's so many different data pipelining tools that one can use. Can you walk us through how you make a decision of what tech stack to adopt as a data engineering team and what are the factors that you look in to guide these decisions?

I'm true believer you don't need to have tons of different tech to be successful. I believe less is more. I choose the tool based on their ability to scale, how future proof they are. Also what kind of level of expertise I have within my team. Right. I don't want to bring literally something that new that my team has no idea how to deal with. And also the price tag of these tools is also very Important you want to make sure that you're not overpaying and you're staying within your budget and cost efficient.

You know, oftentimes as a data engineering leader, when you're making these decisions on a tech stack, you mentioned simplicity of tools and like not having a lot of tools. Why do you think a lot of data engineering teams tend to fall in the trap of buying so many tools and getting so many tools and like how do you avoid that? As a data engineering leader, it's very.

Important to choose the stack that easy to maintain and adapt and also easy to find. The talent will actually support it moving forward. So you don't want to have that tool like multiple tools that do the same thing because it's just waste of money. Right? So two things to highlight here. The tools that easily maintain, easy to adopt the tools, you can easily find the talent and you don't want to have the tools that overlap like doing the same thing.

So let's switch gears here and talk about what makes a data engineering team value driven. Right? You mentioned earlier in our discussion, when talking about what makes a data engineering team successful, it is that it focuses on value and is results driven. A big risk that we discussed behind the scenes is that data teams can generally fall into the trap of building shiny toys that generate little business value but are really exciting to put on a resume. Think like, you know, it's deep learning model or this, you know, machine learning pipeline that doesn't necessarily drive a lot of business value. Maybe walk us through why this dynamic still exists today and how do you avoid that as a data engineering leader?

As a leader, I'm responsible for bringing visibility to the data engineering across our organization. I build partnerships. I usually build a partnership across the organization with various business teams and functions like marketing, finance, analytics, product. We work closely together on the company's strategy and roadmap. Eventually these teams become my team stakeholders. Data engineering work gets prioritized and aligned based on their business objectives. This approach allows data engineers to understand the bigger picture and also helps to identify company wide challenges and the goals. It also helps my team prioritize its initiatives and that's how I usually ensure that data engineering team produces value driven outcomes.

And you're talking here about building interlocks with finance team, with revenue operations team, with different stakeholders within the organization. What does a good successful interlock look like? What does successful collaboration look like with other teams?

Here you want your business stakeholders to be data savvy, thinking of the resource that can be used in many different ways. That's why it is important to educate and show to your business partners what is possible from the data standpoint. A product manager plays a critical role actually in effective collaboration between data engineering and business. He kind of sits in the middle between technical and business. It's important that feedback loop has to be established between business and data engineering. Feedback is super important as well as a regular check ins. So when you have a regular check ins to see how things are going and figuring out what could be done better, that's definitely improve the collaboration between a business and data engineering teams and it's improved how they work over time. And the other thing I want to highlight it's alignment on the goals, right? So we all align data team and business. So we all aligned and on the same page. What are we trying to achieve together? This means the understanding how the data can help the business and make sure the data team's goal is to make a business better and to be good partner across the company.

You're talking here about building partnerships with all of the company. I'm sure as a data engineering leader it becomes really hard to prioritize what is the most valuable thing I can do for the company right now. So how do you quantify different projects that will deliver business value? Like how do you prioritize the roadmap?

Usually the projects that get prioritized first is the one who actually address critical business needs. Projects that also contribute into revenue generation, cutting cost or other KPIs. It's always good to figure out if there are any low hanging fruits, if there are any something we can do with relatively low level of effort but deliver a measurable value. Any quick wins out there, Any quick wins we can get. If we find any type of quick win, it's a good thing to prioritize that.

And when you look at certain projects, you know a data engineering team does sometimes a lot of that work is invisible, right? Like for example building a data platform or you know, improving the integration of one data set to another. How do you look at the value of these projects? How do you quantify the ROI of these types of projects?

Let's say we implemented a feature that recommends additional products for the customer purchase. Let's say you have something in your cart and we build a project that recommends the product based on the current content of your shopping cart. This year plan to improve a conversion rate. So if you. Right. So like you have a product recommendation and we are committing you to buy another product. So we are trying to improve the conversion rate in this Case we would be monitoring a percentage increase in conversion rate using like various A B testing functionality that how would you quantify? So if we build the product and this project drove increase of conversion rate, that it's considered successful. There are some projects that actually doesn't have intangible benefits. It's very hard to measure. Like for example, let's say data team develops a new feature that provides the customer with visibility to their order step by step progress. Like I'm in retail space, so I'm talking orders and shopping carts. So let's say you place an order but you don't have, you don't see the visibility, how it progresses. Now you have an opportunity to see like that your order has been received, it's been processed, the payment being. So you see the detail, the progression of your order. We cannot measure the benefit of that project, but it's actual. But we know that it will drive a high customer certification. Right? And we can update this information like using different service, different customer feedbacks, but it's actually not intangible and cannot be measured. Like conversion rate, for example.

Yeah, that's great because this is what oftentimes when I think about is that, you know, our data engineering team makes a DataSet available at DataCamp to analyze by business users. How do you measure the success of the ROI of a feature that is dependent on other teams using it? For example? So how do you ensure that as a data engineering leader that your partners are leveraging the newly built data that is now available by data engineering teams?

If a customer data platform has been built and the requirement was to build a unified customer data platform, so working closely with marketing, we can help them build a customer segmentation. For example, we give them visualization tools that they can use to actually segment their customer base based on the customer data platform we built. You empower business folks through education of the data, of showcasing what's possible, and again working together to ensure data is used. Visualization tools a good place to start.

We've been talking a lot about how data engineering teams can drive value and prioritize the roadmap and focus on value driven projects. But I think an important thing to think about here is how you can be iterative and agile as a data engineering team. So being agile is something data teams across the board need to adopt more or are slightly less mature than software engineering in general. So maybe what does agile look like for data engineering teams?

I think it's important to be flexible and adapt quickly. If something unplanned comes up, the Team adjusts and moves forward without creating friction and roadblocks. Collaboration, teamwork really important. So we need to rely on each other to do their part, similar as in team sports. So you have to be rely that your team mates will be doing what they need to do. Business strategy changes frequently. Ability to adapt is important. And flexibility ability to pivot into different direction if need be.

And then we're talking about here being able to be flexible and to pivot. But sometimes when looking at massive projects that data engineering teams undergo, such as building a data platform or building an entirely new data collection pipeline, how are data engineering seemed able to kind of piecemeal these types of projects in a way that incrementally drives value. How have you approached this in the past?

If you have a big project on your plate, I usually look at this as building the house. Instead of trying to build the entire house at once. You start with basic a solid foundation, walls and a roof. This is your mvp. The minimum version of the house that functional and serves the purpose of providing shelter. Same goes to the project. As the project progresses, additional features and functionality added into the solution in small incremental steps, right? It's like adding rooms, furniture and decorations to the house over time make it more comfortable and functional as you go. First would be like first we build the required MVP minimum valuable product. And after MVP is delivered, we start to prioritize additional features based on the value that they will be driving for organization.

So you know when you're building an mvp, how do you ensure that the MVP delivers value? Right? Like how do you define what an MVP looks like depending on the project.

For example, this MVP have to be defined together with the stakeholders. So we collaborated to identify what MVP should look like for a given project and identify what features are required and what features are actually nice to have that can be delivered later. So they get prioritized. The foundation has to build regardless. We need the walls, we need the roof, all of this stuff. But then you prioritize the features based on the way they track the value for the business. Start small, win big. Don't try to overhaul everything at once. Pick a manageable project and break it down into the small short sprints like mini projects. So it lets you see the benefits of agile quickly and make adjustments as you progress.

And given the above, like what is the advice that you would give data engineering leaders here looking to adopt agile methodologies? How would you recommend that they start small in their next big project?

For example, work closely with your product. A product manager is your I would say your business voice. Right? Work closely with your product manager, work closely with your business and define what is the important organization at this particular time. And like I said, start small, win big, one step at a time. And prioritization and value is defined together with business stakeholders and product managers.

Okay, that's really great. You know, we've been talking about how data engineering teams can be agile and can drive a lot of value. And we also talked about how data engineering teams can be results driven.

Right.

And focus on projects that matter. We alluded to this earlier in our discussion about the risk of pursuing shiny toys. And I know now with generative AI we see a lot of hype today about the importance of building generative AI tools and generative AI product features. You find there's an increased risk of building something shiny and not necessarily useful here. How do you balance that? As a data engineering leader, it's important.

To create a vibe and atmosphere where everyone in the data team feels like they can come up with cool new ideas and try them out. But at the same time, we need to make sure the team knows that it's important to focus on ideas that actually help the business. It's like, hey, let's try new stuff and see what works, but let's also make sure it's useful and get the job done. It goes back to result driven. You always want to bring the results. You don't want to just spill your wheels for no reason. You want to bring the result, you want to bring the valuable outcome. Let's try, let's experiment, but let's concentrate on tools and functionalities that actually bring the value to the business we are in currently.

And with the context of generative AI, what is the role of the data engineering team necessarily in building these tools? Like, how do you see good engineering teams building generative AI? What's the role of the data engineering team in building generative AI use cases?

Primarily I see data engineering team responsible of actually bringing the data over from all different sources. Generative AI would require a lot of data from a lot of different sources to be in the one spot for actually models to work and use this data primarily focus of data engineering actually to bring this data over, possibly stitch it together so the models can be used as data for learning and training. So responsibility actually to bring this data over and stitch it together for models and data science folks to use.

And do you find that the data engineering skillset will have to evolve, for example, to build retrieval augmented generation pipelines or to build really specific type of pipelines unique to generative AI use cases. Like, how do you see the skillset of data engineering teams evolve over the next few years as generative AI becomes more prevalent?

What I see is that volume of data changing. So we will need to adapt the skills and tools that can process large amounts of data. And like I said, that space and skills are evolving all the time. And whatever we did 10 years ago, it doesn't make sense today. And obviously I see the same thing happening in 10 years from now. So we need to adapt, we need to learn, we need to stay on top of what's going on in today's data world. Again, large data sets will need a lot of storage and a lot of horsepower to process all of that and build actually something valuable.

Yeah, a hundred percent. I think we see a lot of use cases but not a lot of value yet. And then maybe as we close out our conversations, Leah, what are trends that you're seeing in the data engineering space? You know, as we're talking about generative AI, we're talking about different tools. What are trends that you look in the data engineering space that you're excited about, that you're looking forward to seeing in the next few years or next year is going to be maybe hard to predict by the next 12 months?

I see a lot of chatbots being used. I see a lot of changes in adapting the. A lot of I don't think we are there yet. I think we're just taking a small incremental steps towards AI. I don't think everybody 100% understand what's going on. It can be not 100% oriented to what's happening in the field. I think we just take again like small steps and learn and try different things, see what sticks, what works and try to be creative and don't try to reinvent the wheel.

Right.

If there is something exist, let's reuse it, let's use it, let's see what we can adopt. It's a lot of unknown and a lot to learn still for everyone in.

This space, I couldn't agree more. And maybe as we close out our chat, Leah, do you have any final notes to share with the audience?

I would say experiment, don't afraid to fail, learn from your mistakes and don't forget to have fun with it. It's unknown for everyone. There are not a lot of experts in the fields at this moment. So we're all learning together. And ask questions, ask questions, reach out to your teammates, reach out to your partners and keep learning and don't forget to have fun.

That's important. Definitely have fun. That is very, very important. Thank you so much, Leah, for coming on datafriend.

Thank you, Adil, for having me. It was pleasure.