#295 How To Get Hired As A Data Or AI Engineer with Deepak Goyal, CEO & Founder at Azurelib Academy

Hi, I'm Maggie, VP of Curriculum at datacamp and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead. Covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com data engineering is going to play a very vital or the fundamental role for the upcoming AI. The most two important skills which any data engineer need to have is the Python. Second is the SQL. You should be able to communicate because maybe compared to the software engineer, data engineer has to move to communicate because probably they have to talk to the lot of stakeholders to understand what kind of the output or the results they are looking for.

Welcome to Data Framed. This is Richie. One of the side effects of the AI boom is that companies have realized that they need to invest in fundamental data capabilities in order to have high quality, up to date data feeding the AI systems. In turn, that's generating demand for new data engineering talent. Data engineer role has gone from hot to scorching. So today we're going to find out how to get hired in that role. I want to know what skills are needed, what the hiring process looks like and what you need to do to be successful as a data engineer. Our guest is Deepak Goyal, the founder and CEO of Azure Lib Academy. Deepak built a cloud and data engineering education platform and he's taught over 100,000 professionals. He's got 17 years of tech leadership experience and he regularly advises Fortune 500 companies on designing and implementing scalable real world data solutions using cutting edge technologies like Azure databricks and generative AI. Let's find out how to get hired as a data engineer. Hi Deepak, welcome to the show.

Hello Riti, thanks for calling me in. Brilliant.

So today we're going to be talking about data engineering and just to be clear, what does the data engineering job involve?

Data engineer is like someone who is trying to create the data infrastructure. I would say for example, like you have lots of data. Your front end systems, your mobile app, your retail system, every system is generating lots of data. Now you need someone to pull in all this data, do some kind of an ingestion, cleaning, processing and make it available in such a form that that we can derive the insights from it that could be very useful for the stakeholders, that will be very useful for taking it to the downstream system, most importantly like AI systems. So data engineering is going to play a very vital role for the upcoming AI.

So how does a data engineer relate to say an AI engineer? What's the difference between the two roles?

Great question, because a lot of people comes with this question and this is a kind of a very thin line sometimes you have between the data engineering and AI. So you think like that, let's say like you have an amazing infrastructure right across the us you have roads builder so that you can think about like a data engineering. And the Tesla moves on that like the driverless car is something like an AI part of it. So data engineers build the data platforms, they bring in all the data, clean, transform and make it available like gold layer data, which can be utilized for getting the business insights. And that's where the AI people will come in. They'll take the advantage of the playground, the data playground, which has been developed by the data engineers to take up the data from there and then use it forward. So I would say that like data engineers comes first and then on top of that the AI works start. So that's how you can differentiate the data engineers and the AI engineers.

Okay, so the AI engineering really it's sort of built on top of the data engineering.

So once you have this data, then the AI engineers will utilize this data to do the prediction, forecasting, or maybe generating the new data using some gen AI technique like content generation and all. So that's where they'll start from.

It seems like data engineering is a pretty fundamental role for a lot of organizations. I'd like to know what's the most common path to get into this job? Like how do you get started?

The most two important skills which any data engineer need to have is the Python because they're going to write a lot of logics using the Python. So that would be the fundamental skill. Second is the SQL. So SQL is the heart of the data engineering because you're going to play with data and fortunately almost all the tools which you find across data engineering, you find that in one way or other way they support the SQL. And SQL is easy to learn as well. So these two skills is something which is going to be the foundational one, which is going to be the, the root. Once you have built that skills, then on top of that there are further skills which someone needed to learn to become a data engineer, for example. After that they need to Learn a kind of an ETL tool that helps to pull in the data. Then if they are playing with a huge amount of a data, they need to bring in the big data technologies like Hadoop, Spark, Pyspark. Maybe they need to learn the new data warehousing tools like the cloud ones like Snowflake, including the data breaks in. That's the main skills which someone need to learn. Besides that, then you need some, I would say secondary skills like how this code deployment will happen, how to do the cicd, how to do this performance optimization, doing the data governance and all that also then add up as you move forward or upward into this data engineering journey.

Okay, so actually lots of things to learn there. So it's interesting that you mentioned that. So SQL and python all obviously like two main data programming languages. These are the important places to start. So after that you said it's about data processing, big data processing technology. So Spark and things like that and then the sort of cloud data warehouses as well. Okay, all right, so is there a particular order you need to learn these like what do you do first?

As I said first this SQL python is basic because without that probably you cannot learn most of the ETL tools and most of the big data analytics tool. So once you've done that probably you will not be ready as a data engineer. Then you definitely need to learn at least one ETL tool. Now we have a variety of ETL tools in the market. It could be like an airflow which is an open source. If you go towards the cloud, then in the Azure world you will find the ADF Azure Data factory. If you are more focusing towards the Amazon or the AWS then the tool would be there is a AWS glue. So once you have the SQL python ready then you need to little bit pick a stream around that. You want to go towards a one specific cloud or you want to go towards an open source system. So once you pick that branch in that then you have to learn one ETL tool. So that's one ETL tool is needed. Okay, it could be either on the AWS track, a app Azure track or it could be an open source track. Then after an ETL tool the next thing which you need to learn is the big data analytics tool. So in the big data analytics world the core skill is the Spark Pyspark. Now the same thing happens is like Pyspark you can do on your own. But these days people uses the databricks. So maybe again if you want to go into the databricks world, you can choose Databricks if you want to go towards the Microsoft, which also 90% of the work is like the Pyspark only, but they have recently launched a very popular service that is a Microsoft fabric. So then in that case you need to learn Microsoft fabric. Otherwise if you go ahead with the aws, then AWS also has the AWS Databricks. So you can learn that big data analytics tool. So in short, SQL Python, then one ETL tool based on your track and then one big data analytics tool. And then later on you can slowly, once you are ready with this, I think I would say that like 90% you are ready for the data engineering job. You have to just add the remaining skills like cicd based on the kind of a tool which you have select and add little bit of the more skills around, some behavioral skills and all and then you are good to go.

That actually sounds very straightforward. Learn some Python, SQL, learn an ETL tool, learn a cloud data warehouse and you're kind of there. All right, so beyond these sort of technical skills, are there any soft skills that you think are important as a data engineer?

I would say that any job in today's time must need to have a soft skills. So data engineering is no exception to that. The first and foremost is that you should be able to communicate because I mean maybe compared to the software engineer, data engineer has to move to communicate because probably they have to talk to the lot of stakeholders to understand what kind of the output or the results they are looking for. They have to coordinate with the data analysts, they have to coordinate with the AI engineers. So they might have to communicate a lot of times with the administrative platform admins and all. So they definitely need the first important thing that is a communication skills that they would be able to put their point forward first. Second, the behavioral skill, which I feel is that they should have a problem solving skills. A lot of time you get into the stuck that you build some data pipelines and the pipeline is taking huge time to get finished. Or maybe sometimes you develop everything right and suddenly it happens that the data analyst team or the stakeholder comes to you or business user comes to you and say that okay, we are not getting this specific values into our spreadsheets or into our dashboard which was there. Then you need to be smart enough to understand and try to put your critical thinkings into the play and see that where is the problem. Then later on you might find that okay, the incoming data has itself a problem which is not populating this specific information. So you need to have this critical Problem solving skills. That is something I would say must have it. And third, I feel in one of the most important in general in today's time is the adaptability and learning capability. You need to be someone who is continuous learner so you cannot stop because we are into the era where everyday new technology is coming up. So you need to keep on learning new skills and you, you need to be open for that. So this is what something the soft skill people look for towards or the kind of a behavioral skill people look.

For is kind of interesting. The idea that you will get stuck on a regular basis because things break and sometimes things get hard. So having that ability to overcome being stuck is an incredibly important sort of I guess is it sounds like as much as an attitude thing as a skill there.

I think absolutely. See when I personally hire, I have a long history of hiring people. So I look for this very important skill that as you said about the attitude because I mean technology can be taught okay, but the attitude is difficult to develop. So that's create a difference. So we need to look for that and every recruiter looks for that.

Just going back to the thing you mentioned before that you need strong Python SQL skills. Are there any particular things you need to know around programming? Because I mean these are big languages, I mean Python in particular you can spend a whole lifetime learning. Are there any like specific things that are useful to know?

The good thing is that there's a lot of AI involvement comes in so a lot of functions nowadays it's coming up with auto return for you. So like the IntelliSense has been developed a so strong that we are slowly moving into the low code or no code area. So although still the Python is needed, I would say that people get confused when we talk about the Python from the data engineering perspective and the AI perspective because in case of the Python for the data engineering perspective you need to know I would say the basic concepts of the Python like how to declare variables to do the control loops, if else. So those kind of stuff good to know functions, little bit library is good enough. But if you go to the AI side then you need to know a lot of things into the Python. I mean you have a Scikit learn, you have a Pytorch and all. So don't get confused. If you want to build a career in the data engineering world then the Python those data science library is not something needed. You need to be covering the basics Python where you are good to write the code and then you would be smart enough to maybe use the Pandas or little bit of a numpy, that's it. If you cover the Python up to that level, you're good because rest of the work, remember, is done by the Pyspark. Because your Pandas is not a distributed computing environment, so they are not smart enough to handle a huge amount of data. But the Pyspark would be so majorly going to write the PI Spark. So that's why the fundamental Python is good. If you come to the SQL side. SQL side. I think normal SQL people understand the only exception to that normal SQL would be here is the Windows function. So you need to be a little good around the Windows function, like the RAM and lead lag and all. So those are the extra things. If you're not aware of that, someone has to cover that.

Okay, so on the Python side it's incredibly interesting that you don't need all these data science libraries. So I think, yeah, if you're a data scientist, something like Scikit Learn is pretty much core functionality there. It's something that's very essential. But as a data engineer, it's like keep it simple, just stick to like the core sort of syntax a lot of the time.

Yeah, because what happens is this is one of the biggest misconception. See, over a year I usually train more than at least 10,000 people to become a data engineer. And every time people come to me have this common misconception that they need to be very expert in the Python. But reality is not like that. So because majority of the work you're going to write, all the complex logic you're going to write using the SQL itself. So if Python, you have the practitioner level knowledge, it's good enough, you don't need to be expert of the Python.

So basic Python skills. On the SQL side, you said you also need window functions. Now this is interesting because this is like one of the first sort of hard concepts that people come across when they're learning SQL. Talk me through, why do you need window functions?

See, many times you come across a situation where let's say somebody is trying to identify the running total or maybe somebody is trying to find out. Let me give the top five product which is get sale on a monthly basis or something like that. So you get into a situation where this kind of functionality is something it's important to learn. Also everyone who is learning this data engineering, the simplest goal they have in front of them is going for the job interviews. And these questions are the most favorite for the interviews.

Interesting. Okay, yeah, so since you mentioned interviews, we talked a lot about skills. So let's talk about getting hired in terms of what hiring managers want. I guess to begin with, do they care about qualifications? If you want to be a data.

Engineer, most of the time not. And honestly it's literally based on the region. I would say in India, if you talk about the India, few people, you find it like a very stereotype, they might look for that. But if we talk about the majority across the globe, like in the us, uk, Australia or the different parts of the world, generally people are open. So I feel that most of the times which I have seen is like the hiring managers are not very much behind the specific degree. But again, if you are belong to the computer science, it's always, always a plus in cases when you are not from the the engineering background in those cases or not from the computer's background or maybe like a non IT background. I would say in those cases somehow you need to showcase your credibility because the computer science engineer somehow has a credibility that they have their IT degree. So in that case, I mean you have to play a little bit different way that you do some kind of a certification and you have some projects to showcase and all and that's how you know you can project yourself and you can justify that. Yes, although I don't belong to a specific category of folks, but still I am, I'm someone who knows all this and I can handle this job.

Okay, so you've got to demonstrate your computing skills, your programming skills. So if you've got some sort of, I guess STEM degree, then that's going to help. And it's only if you're coming from a different background that you're really going to have to try hard to showcase that you do have those programming skills. Okay, so you mentioned projects, talk me through that a bit more. Like I guess this is going to be creating some sort of portfolio. What, what might you need to put in that?

In the portfolio case there's two basic things we need to understand that basic challenges the portfolios. If you are someone who is working professional, then probably portfolio creation is not that easy. The reason being would be is that you might be working for your client and you cannot share the data so you cannot make a GitHub repository and saying that okay, this is my GitHub repo and like I put all the code which I have done or all the SQL logics which I have written in available there so you don't have that leverage to do it. So in that case probably in your resume you might be putting it and that could be considered because the hiring manager also aware of that. So for them probably at least as far as the data engineering is concerned, it's tough to make that the portfolio. Now if someone who is newly entering it into this data engineering field, what they can do is they can pick some of the we call like a POCS or some kind of a capstone projects that they can build in and they can put it onto the GitHub showing all the codes and all the logics they have written for it. So that's how they can think of building the portfolio. And if we talk about specific about the projects. So the project should be somewhere like creating like a data lakehouse system where you have some medallion or a three layer architecture. That's a very popular framework at the moment and most of the organization is implementing that. So if you have a one of a project around that that going to, you know, put a lot of value to your portfolio then second would be like if you can bring in something into the real real time streaming and all that is something recruiter love to see that.

So just having real world examples of hey, this is like something that's going to be useful. And then I guess it doesn't have to be huge just to show off that you do have these skills to be able to create something once you get the job. Okay, are there any other ways you can make your profile stand out if you're applying for data engineering jobs?

I mean there could be multiple things. So the first is your resume. So you cannot skip that. So make a killer resume. I would say the resume when you're going to make and resume, try to put it in something which is quantifiable. So rather than just saying that I work for some XYZ project and I work exceptionally there, maybe that not going to work. So maybe try to quantify that work. Maybe you start seeing that, okay, I work for this XYZ project and I built the pipeline that has improved the turnaround time by 30% or something like that. So you need to quantify it. Maybe you need to start quantifying there like how much data you size you have handled. So rather than just saying that I worked on a huge data and handling that start saying numbers say that okay, I was handling the 100 terabytes of data on a monthly basis. So that will going to put something that people can quantify. So that the first thing would be to make yourself stand out having a killer resume. And killer resume will have something like this, that quantifiable thing. Second probably is as we were talking about it, you can always add a certifications so you can think of doing some certification based on whether you are in the Azure track or AWS track or maybe databricks or maybe snowflake. So whatever, you are picking the technology based on your liking. So in your space at least do some kind of a certification so that something is making you stand out against all your peers. The third thing would be is being more active over the social media like LinkedIn which is specifically for the professionals. So maybe that catches more eyeball compared to others. So you can always say that okay, I'm quite active and I'm sharing what knowledge I have and it's kind of, you know, building your online portfolio every day. So maybe you can do that. Lastly, I would say like if you are good at writing or you know, speaking skills or something, so maybe you can think of writing blogs, making some good informational YouTube videos or something. So in short you just share your knowledge. So somehow that will make your portfolio saying that yes, this is the guy who knows something. That's why he is sharing with the world. So overall this whole bouquet of things, if you do it probably you're going to, you know, get yourself into the top 1%.

Yeah. So really just like nail that resume and then that's going to really sort of help out, make your application stand out. Okay. All right. And yeah, beyond that it's the stuff we talked about is getting all those qualifications and the portfolio demonstrating that you really do have those skills. Okay, so talk me through what does the hiring process involve typically?

So hiring process in general always like very big based on company to company. But it start with you apply for the job. The moment you get shortlisted then you probably get a first the screening call from the hr. He might ask you some basic questions like what do you do, what total years of experience you have, how's early or how soon you can join if we give you an offer and all. So those kind of an initial screening they do it and then we get into the technical discussion. So the second round would be the technical round. So it will be generally it's a intense technical round where they're going to try to ask almost everywhere from the start to the beginning. They might ask you all the tough questions around the technology based on your experience. Maybe sometimes we have a second round of the technical as well. So overall like third round I would say so that will be another technical discussion round. Sometimes some organization call it like a system design around where probably they although system design Is more or more popular in terms of the software development engineers like SD roles and all. But in some cases in the data engineering world some of the organizations still do this system design. Or maybe the second round of technical questions discussion where they are trying to ask more of a real world scenario. So they probably give you the scenarios like we are having 100 terabyte of data and you have to design the pipeline. So how you would design this pipeline and you need to ensure that if something is going to break in between then how we can recover that situation. Or if that you are getting a late data how you're doing the backfilling in some situation. So they try to twist the question around these different use case scenario or most of the failure scenario. So something which you need to prepare yourself for these kind of a scenario based question not mostly that is not going to be straightforward. There is something is more of like you might have experience. You might also get some questions like okay, what are the different challenges which you faced in technically and maybe tell me the most challenging problem you have solved technically. And then you have to explain that scenario like that. So that is the second technical round. So overall third then fourth would be the behavioral or the managerial realm where like they asking the question like to just check like whether you are right fit for the organization or not. Maybe from these they try to test your soft skills as well. They try to check the the compatibility within the team that okay whether you're going to be compatible for this respective team or not. Whether you have a right attitude or not. So all those the soft skills that okay, they give you some conflicting situation and say that okay, how you resolve the conflict in this situation or if you were there at place how would you have handled it etc. So those kind of a behavioral questions or maybe altogether some problem solving skills. They're probably going to ask in that. So more of like a little bit of non tech interview in the behavioral round. And then mostly then would be the next last round would be the fifth round would be the again HR discussion where they probably talking about the compensation and all. So I think if someone passed through that probably it's time to roll off the offer later.

So it sounds fairly comprehensive. I think this is a fairly standard for a lot of tech roles is you have some sort of a screen and then it gradually gets more technical and then you have an interview with the manager. So in general do you have any tips for how to pass interviews? Like what's going to put things in your favor?

So non tech Side I think the first and foremost is like be genuine. So it's tough sometime to twist your personality. So be genuine the way you are. And I feel that most of us are in general are good people. So probably that will show up there as well. Coming to the points, you should always have a one killer resume. So that is always going to help. Then next important tip, I would say that if you have the attitude of a learning so you need to keep yourself open because the technology in which we are right now is changing at a very fast pace. So to keep up with that pace, maybe you are working every day, day in, day out. That's maybe if you want to stand out your chances and you know, tips to show your skills compared to others. Probably you need to open for learning. So keep on learning new technologies. So keep your eyes open. That is another thing. The third thing I would say that have more networking done. So maybe like through the LinkedIn or somewhere, go out and talk to the people, build your network. Because a lot of jobs get filled. I don't have the exact percentage in front of me but if I'm not wrong somewhere I've read it around 30, 40% of the job has been filled through the networking itself, like through the references and all. So there's a huge job get filled through that references as well. So build your strong network that also give you an added advantage outside your technical knowledge and maybe if you can have the credibility in as well in terms of the certifications and all. So that will help you get through the job easily as far as the non tech is concerned.

I quite like that idea of being authentic. It just seemed like sensible that you don't want to twist your personality and pretend to be someone else. Because once you get the job people are obviously going to find out who you are in the first place.

One question you can do it, but they throw you another question probably where you show up your real face so then it will be conflict. So maybe even they liked you by the first question. They start hitting you because they feel like that you may be pretending a lot of things, not just this thing. So I personally feel sometimes that being you is always easiest for you.

I guess it's a good idea unless you have a really awful personality, in which case probably don't bring that to the end.

Generally don't have it. Trust me, most of you don't have.

Okay, yeah, bring the best version of yourself. Maybe not just show quotes and all in the interview. Okay. And the other side of things was the tech side of things. So what tips do you have for the tech interview?

For the tech interviews the straightforward question is fine. I feel that somebody should be very well adverse with the performance because in the detangling world the performance is the key questions which you can expect. Someone would be very interested that how you handling a huge amount of data. So think about how you're going to handle the performances. Second probably is the failure situations. So happy scenario. Everyone is okay that you'll be able to write the happy scenarios. You probably they are more interested in understanding that how you handle the failures. So think about that. Okay, your normal pipelines, your normal logic works pretty well now if it starts failing, how are you going to tackle that? So that would be something which you can expect. So that is something which you should prepare beforehand so that you don't feel like a surprise into the interview performance failure handling the failures, youth size of data handling. That is again kind of a performance itself. Maybe this is a one very very important question which I seen the most of the time is asked like tell me the most challenging situation you faced in the technical interview. And people get caught at this level and people feel that they really have to talk about the exact challenges they have faced it. So they need to play smart there probably they need to talk about something which they are most comfortable with it. So they bring in a one scenario which they are most comfortable in talking and most comfortable in explaining. So bring on that and showcase it like the challenge which you have solve it. But maybe it's a challenge for them. So being you are strong in that area it will be an easy task for you. So bring it something example from that. So beforehand prepare for such kind of an example. That is something is going to ask in general maybe in on the SQL side window function somebody should need to prepare because window function is something people find it a little hard. So some question might you can expect on the windows side windows function site. And most of the people get stuck around this. And lastly like any new new feature got added up. For example if you're going for a databricks interview then just go and read it. What are the in last six months. What are the new features got added up. So people get. People are sometimes are interested that do you know that new feature in. So they just want to check like whether you are updated or you are someone who is keep learning new things or not. So just do that and then I think probably that will be your ticket to another job.

Okay. I do love the idea of preparing things beforehand because certainly I spent A lot of my time fumbling over, trying to explain technical concepts. So I think, yeah, preparing what you want to talk about beforehand, especially if it's like talking about a project that you did a year ago. You've done a few projects since then. You probably can't remember, like what the hardest part was off the top of your head. You're going to want to write that down and just practice saying it before you go to interview.

My ninja tip always for my students, and I think for all our listeners here as well, would be, is that even when question comes to you and interview, even if you know the answer, pretend like that you don't know the answer. So the question throw to you should not jump and answer. Take a pause and pretend it like that. This is something new question is offering to you and then answer slowly. So it feels like the interviewer that, okay, he has asked us some really a tough question that works pretty well many times.

Yeah, just take a moment to think before you engage your speaking muscles.

Because it's a human psychology. I feel that interview is almost like a fight between the two minds. Right. Everyone wanted to win. Like the interviewer wanted to win and feel that, okay, he has asked one of the toughest question to the interviewee. So just let them let him pass and let him win. Feeling that, that, okay, he has asked really a tough question. And then you take a pause, you feel like, okay, okay, I never heard about this question. Let me try to. Try to answer like that. So that brings kind of a perception in the interviewer's mind that, okay, he knows something because this is not something the question which he is already aware of it because that will decrease your value in general in their mind sometimes.

Okay. Actually, so just related to that, so you mentioned sometimes you get an interviewer who just wants to ask the hardest question just to kind of show off.

Absolutely.

What do you do when you have a difficult interviewer? Yeah, yeah. What do you do when you have a difficult interviewer?

If you have a difficult interviewer, the best thing is you need to be polite. Be polite as far as possible because there is no point in arguing with this person because he's eventually a difficult person. So. And he's always having an operand because he is the interviewer and you are the interviewee. So anyways, because he has the final call right to take, so there is no, no point in fighting. So I would say put your guards down and saying that, okay. I mean, be polite and answering. Maybe he will try to counter you sometimes that, okay, why this why not? And he may have a very rigid personality in that case polite and put your point forwards with the facts and figures. So if you with the factual figure then probably he will left with no room to argue with you on some of the stuff. So put your points, you know very factually correct. For example, if say that okay, you say that okay how you improve the performance then you will say that okay, I'm going to improve the performance by doing the caching. And then you see that not just you just let him complete. I mean put the figures as well that because if you cache the data because your data gets saved into the main memory and to get the data to read the data for the processor from the main memory is always going to take a less time. It has a low latency compared to the data which is going to be read from the hard disk. So that's why to improve the performance it makes totally sense to put your data into the cache. So something like that you. You. You put your answer and add little more extra description around it with which will put your points forward with some facts.

All right, so I'd like to talk a little bit about the use of generative AI because obviously generative AI can help you solve many problems during the hiring process. Now I know Anthropic quite famously doesn't allow the use of generative AI during their interview process even though they're the creators of Claude. Do you have a sense of when you might be allowed to use generative AI and is it a good idea.

To be honest at this point most of the organization is not ready to accept that point. Still at this point a lot of the organization you find it which even does not allow within the organization itself not allow them to write the code using the some charge party or any AI based tool. So I think the world needs some time before we make it little comfortable. I mean forget about the agent or the the AI you think about in today world still people are not allowed to do the Google in the interviews. Most of the interviews you see, right? I mean although like we when we work as an engineer I'm working from last 17 years in IT industry since day one of my engineering I'm using the Google every day almost while writing my code and all. So there is no shame in doing a Google. But still in today till this state most of the organization is not ready to allow you to do the Google. So I think expecting that they will allow some AI tool to to use it into the interview. I doubt that they will be open so fast, fast. Maybe some of the startup could think of doing it maybe because they might be little open minded in thinking that okay, once a person comes in and probably we need someone that who is going to use this, all the available AI tool in the world and build some software for us and we are okay with that. So I think those startup mindset folks probably can start up with that. Ideally I think for the bigger organization I feel it's still. It's too tough. It's too tough.

Okay. So really it's going to depend a lot upon the organizational culture and it's probably something you're going to want to ask about. Maybe in that first interview is like.

I think the acceptability is not right at the moment. I think it might take some time because I mean in day to day, I mean think about day to day like when we talk to the people or let's say you go to your manager and say that okay, I have written using the ChatGPT probably he doesn't feel comfortable. People feel little bit negative side in saying that okay, maybe you, you don't need it, that's why you do it. Although I feel that I, I was not a part of that journey but maybe I feel that like 20 years ago or 25 years ago when whenever the Google came in like and when people were using the Google help and writing in the same feeling might have it. So I think that acceptability is yet to come. So hopefully after that they might be open for the interviewee to start using the AI.

So one more question on hiring is suppose you get an offer. How do you go about negotiating your salary or pay package?

The negotiation is a top skill. Everyone should have it because it is not something which is needed in your job roles. It is like a day your negotiation skills is going to pay you big time. So that is something everybody should have it. But if you come to the point of like how you negotiate in terms of the salary when the HR is asking to you the one thing I would say that before getting to that point at least we should have our homework done. So we should be very well aware of that. I'm applying for let's say a senior data engineer role. So I should be very clear on that. Okay, that for senior data engineer role in this specific organization or in most of the organization of this level and in this specific country and in this specific city, this is the range. So at least I should have an overall range in front of me. Okay before negotiating because that's how I can get an idea that okay, this could be the right price or not. So first, that homework needed to be done. Second, once you have done that so you have some range is in your mind that or you get to know that, okay, like let's say 100 to 150k is an average salary for, let's say for a senior data engineer, just an example. Then we you get into the, the negotiation round or that discussion, then rather than asking for something directly that okay, I'm looking for a hundred or I'm looking for 150, maybe you start seeing that like, okay, I have done some research and I feel that the knowledge which I have it and the kind of a profile which I have it, the market standard or the industry standard is up to this range. So then you talk about that range so that you've opened up it. So rather than just taking out a one figure, you have opened the range. Now most of the time there is always a scope on your initial offer. If HR is offering you, let's say 100k, he always and always most of the time is a cap of at least 10% plus doing on top of that. So it's there. It's only that few people because of some fear that probably they're going to lose the job if they say the moment they ask to her, I mean ask little more and then probably the HR will say no to it. I feel that that's a little bit misconception because most of the time this does not happen. Okay? Because think from the organization perspective. Organization has already spent a lot of time on you. They might have taken three, four, five technical rounds and after that you got selected. So in their eyes you are a right candidate first. So they have already put in a lot of efforts on you. And if you feel that just asking for money they will leave you, probably they're not, maybe they will say that, okay, this is the best we can offer. Please tell us that, okay, whether you are okay with that or not, but asking more, I think it's not a problem. It's not going to lose your job. So you should give a range and try to ask to the top of that range. If you feel that that discussion is not going very well, then you say that, okay, this is the top of the range which I'm expecting, but I'm a little flexible because I want to work with you, something like that. And then you do it. So at least it's right that probably you not accept their number. They're not going to accept your number. But it is midway Something you can break the deal and that would be plus for you. So get into that mode. I feel that that is something that's going to help you a lot to negotiate it in better manner.

I like the idea that there is a lot more upside to negotiating than there is like downside. So yeah, the worst that's going to happen is they're going to say no and give you the same offer as before. They're not going to completely withdraw their offer unless you are like really rude about I guess how much you're asking for and ask for double the salary or something like that.

And also just to add one more thing, position power get multiplied the moment you have a multiple offers. If you have a multiple offer then definitely you can negotiate very hard. So that is also the thing which you can think of it like the moment you have a one offer then you can also put that point very, very smartly there. Okay, saying that okay, I'm looking for this some X to Y range and I'm already having in the final round with another big shot company. So I'm expecting an offer from there as well. But although I'm still love to join your company, I get the higher side of this range like that. So you can smartly put that point as well. Multiple jobs and the moment you go to one offer, the next offer is going to be, you know, you have a very, very strong negotiation power with you.

Brilliant. Okay, so suppose this all goes well and you get the job. What's a likely first project for a data engineer?

I think if you are a experienced professional then definitely the project would be like whatever their highest priority project to be honest because they might be hiring you directly for some of the projects. So most of the organization will not going to be so choosy on what to give you and what not to give you. So because they already have something in front of them and they probably is hiring you directly for that requirement. Maybe it could be your some data engineering building, the data Lake House project or maybe like streaming based project could be depend whatever the requirement they have on the second side. Let's say if you are someone who is a beginner or a fresher for the fresher probably there is some chance that they might not directly put you into the one of the critical project or maybe the most important project for the organization, they might start you with little the smaller project with some smaller client or the project which they've started or trying to do some kind of a POC and all. So that's where probably you can place in in case of the fresher but mostly in case of an experienced professional. Probably they know that you're going to get fit there. Probably that could be the project which you going to get it. So they'll not be so choosy in that case. Because as an hiring manager I would always try to hire the best people for the best project for whatever the project I have. So they're not choose like that mostly.

Okay. So it's going to depend a bit on seniority. I say a data camp, there's like a policy of like anytime someone gets hired, it's like make sure that they can have like an easy win in the first week just to make sure things come along smoothly.

But generally data engine project is like a year long or a six month, eight month long project. So you cannot run it. Someone gives so much room to, you know, breathe.

Absolutely. Okay, so more senior kind of the.

The bigger and higher priority in that case. You can think of that. Let's say if I have a agile based project where I'm going to run multiple sprints, so maybe for initial one or two sprints they might not give you a lot of work in that project, you might get assigned to the bigger project, but for initial spins you get a small amount of work so that you get adjusted to the environment, you get adjusted to the work culture of the project and all. And then slowly and gradually they will increase the load on you.

So I'd like to talk a bit about what happens after you get hired and how you keep up to date. So as you mentioned, technology's moving pretty quickly. There's been big shift like for example, like Microsoft launching fabric that's kind of like fairly new and evolving rapidly. How do you keep up to date when the technology is moving so fast?

So I think the first thing would be is follow Richie. So Richie has been. So I mean that's an easy way, right? You are going office, you can just switch on your the podcast and you can keep listening. So a lot of people might have the problem that they don't have a time. Right. So maybe the Richie's podcast could be one of the best. So you keep on bringing a lot of new folks and who are like a leader into this specific technology and the stream. So listening to those people always give NFA advantage. So that is one way is like if you don't get the first thing like people if can get set and do it, that's the best. But if they don't have that leverage or they don't have the time right. Like following these kind of a podcast could be one of our easiest way. Probably I would say that people can do it. So that's a one way to keep yourself updated. At least you get to know the new things and maybe this is something is coming in which is important for you, which is not not important for you. So you the first level of iteration is definitely going to be done here itself. Then second, I would say that if someone can get time probably they can take the pointers from these kind of following the leaders, listening to podcasts, following people on the LinkedIn and all they get that pointers and with that pointers they can go further deep down sit and they can read a little more about it. If they get some time to do hands on amazingly well they can do that. If you feel that like you still need to be someone who needs some kind of support to make you aware of everything. Maybe they can think of taking some online courses, a quick short courses where maybe go to the data camp, some quick course of the latest technology that could also help somehow to keep you updated with the right set of technology in an easiest manner so you don't have to look here and there, where to read, what to read and all right, so they can just simply go to the data camp and then pick a right course and probably in next two, three hours they'll be ready with the best summarized content. So that could be another way maybe then if you have more time, you may be reading some blogs or following some specific art. Let's say you work in the snowflake or in data warehouse, I mean data breaks and all. So maybe you can follow their blogs and read some announcement, whatever is going on. So these are some of the ways probably where you can keep yourself updated.

It's good that a real mix there of like maybe learning is as simple as just reading a blog or listening to a podcast. It doesn't have to be formal learning all of the time, but if you need to get into a bit more depth, then that's when taking some courses is going to help out. So I guess the tricky bit with that is you've just got a new job. You're probably swamped with learning about all this new stuff in your new job. How do you find the time to learn and work at the same time?

That's one of the toughest question. The time is critical, but as I said like finding time, it's more of like your priority. So maybe I would say that make some routine. So as we were talking about it right that getting a time is tough like but at least let's say you are traveling so at least you can listen the podcast. So at least the bare minimum that somebody can think of doing it. Or maybe I would say like if you're working at least make a calendar that okay, once in a month I have to sit for two hours. Nothing like that. I'm like I'm just fixing my calendar a month prior that okay, I'm going to sit for two hours a month in my calendar. Maybe I cannot do it for two hours in a straightforward. I could cut it down like a 30 minutes. Thirty minutes. Thirty minutes. Thirty minutes across four slots. Or maybe initial maybe forget about two hours. At least block an hour or 30 minutes of a two thing. I mean two calls you can block for yourself, two calendars and then you make a habit that okay, I'm going to go and sit and write it. Maybe like you do a lot of social media, right? So maybe on the social media itself maybe you. You love to the Instagram to or the TikTok to watch the videos and all maybe at least start following some of that tech influencers as well. So maybe while you're surfing it for a 10 seconds, 20 seconds, 30 second they will talk about something and you might learn something new. Probably because these influencers are the one whenever something new get launched probably they are the first one to start talking about it. So maybe these are not directly but indirectly make you little bit aware of the new thing which is going on in cases when you have zero time dedicatedly for learning stuff.

That's actually quite a good hack for your social media is just make sure that you are following like some relevant sort of work people as well as just like the fun stuff. Yeah, yeah. And then yeah maybe a 10 second TikTok video is better than learning nothing at all. You're gonna certainly find out what's coming soon.

Yeah.

All right, brilliant. Do you have any final advice for people who want to get a job as a data engineer?

The final advice Python 1 to ETL tool PySpark. I would say that master this. Second, if you can add on some of the certification. Well, if not that's completely fine. At least you should have a real world knowledge of hands on practice. So do focus on more doing the hands on because that is more needed even than your certification. Do that. Third, I would say that that is one of the biggest hack I would say that is whatever you are learning, try to teach in some way. So maybe by writing blogs, by posting on social Media by posting on LinkedIn or something like that, that will build your portfolio as well. Secondly, probably whenever you try to share your knowledge in in any of these different ways, that helps you to understand the things much, much better way. So probably if you are learning, if you are sharing and if probably that eventually build your entire ecosystem that will pushes you for your best and that will going to benefit you a lot.

Brilliant. So I guess, yeah, get all those tech skills beforehand. Make sure that you've got something to demonstrate your value, the certifications, portfolio, things like that, so you can get the job and then yeah, just keep learning. All right. It sounds so easy. Nice. All right, super. So just to finish, I always want follower recommendations. So whose work are you most excited about at the moment? Who should I be for following on social media?

As I said the first one be the Richie. So go ahead, follow his podcast. I mean I come across a couple of months back, so now I started listening to him. So that is the first recommendation I would say the second one would be myself. So you can keep following me. If you are someone who is talking about the data engineering that you can do it. Third, I would say Zach Wilson. So Zach Wilson is stuck a lot around the data engineering so you can find him on the LinkedIn. So he's also making a very good content on the data engineering side. You can follow that. I'm personally someone who is following him a lot. So that would be a good people to follow.

All right. Yes, follow. Excellent. All right, thank you so much for your time, Deepak. It's been great chatting with you.

Thank you. Thank you so much. See you.