#249 Towards Self-Service Data Engineering with Taylor Brown, Co-Founder and COO at Fivetran

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with.

The rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated.

The best learning content to help you.

And your team stay ahead, covering everything.

From data and AI literacy for busy executives to cutting edge generative AI for.

Developers, and everything in between.

Whether you're upskilling yourself or your team.

We make learning practical, engaging and flexible.

It just works. Power your data and AI transformation journey on datacamp.com I think this is the holy grail, right? This is the democratization of data. Like anyone can ask any question. That I think is the big promise. The challenge that I've seen in practice is even when you implement this, people don't always know what to ask. They don't know what questions are supposed to be asking. So there's sort of one side which is, you know, the human element of it. And I think AI is going to play a big role in this foreign.

Welcome to Data Framed. This is Richie. In the last few years, it feels like data analytics has become more well defined and standardized. There are a few great business intelligence platforms, and how to make a great dashboard is well understood. On the other hand, data engineering is more complicated. Data is generated from an increasing number of sources in different formats, and there's a plethora of tools to organize them. It's far too common for data to be stuck in silos where the right people struggle to get access, or the data team can't find the right data sets, or the datasets aren't documented properly. This has been a problem for years, and there's a modern data stack to solve it. But maybe that's out of date because people are already talking about a postmodern data stack, and there are several governance models and organizational structures to choose from. It's such a muddle that it's hard to know where to invest. Today I want to get some answers to how we can fix all your data infrastructure issues. Our guest is Taylor Brown, the chief operating officer and co founder at fivetran, the wildly popular data movement platform. Having spent more than a decade building FiveTran, Taylor is one of the most influential figures in transforming the way organizations manage their data infrastructure. At this point, he knows more than pretty much anyone about getting your data sets from their sources to the right people. So let's see if he can help us figure out all those data messes. Hi Taylor, thank you for joining me on the show.

Richie, great to be on the show with you today.

So I want to talk about problems to begin with. I'm sorry, what do you think are the biggest challenges in data engineering right now?

I think there are a few massive challenges. The first one is just, there's just so many new tools that have come out and there's so much emphasis towards AI and AI tooling that it's very difficult to understand what should folks be doing, what should folks not be doing? To answer a little bit more about what fivetran does, just as a quick background, I'm the co founder, chief operating officer of 5chan, and we move data for companies. We're the global leader in data movement. You know, loading data for Fortune 500 companies all the way down to tiny startup companies such as, like OpenAI and Snowflake Databricks, Morgan Stanley NAB and thousands of other customers. And so I often think about the world in terms of data, data movement. And so probably a lot of today's conversation will skew that direction, just so the audience understands that background. But yeah, to answer your question more directly, I think it's just this explosion of tooling that's happened over the last few years and understanding like, where should we invest? Where should we not invest? What are the right things to build on top of? Because ultimately, data is like one big stack, right? And you have to build on top of each layer, and the layers that you're building on top of, they have to be layers that can survive for a long time. That's the biggest problem that I would.

Say that's absolutely fascinating. There are so many tools, and it's something we struggle with at datacamp as well. It's like, which tools do you teach? Because are they going to be around in a few years? Which ones are the most important ones? So do you want to elaborate on that? Like, how do you go about deciding which tools you need in your data stack?

When you think about the overall data stack, the biggest problem is basically like, every company has a ton of data, and now that data is all over the place. And so at the end of the day, you want to use and utilize this data. And that's the overall goal of building a data platform. So you're using it for AI, you're using it for. Now that's the biggest one, AI ML, you're using it for bi, for building applications. Like, these are some of the use cases. But then the question becomes like, okay, well, where's all this data? Why is this data Everywhere. How do we get access to this data? Do we centralize it? Do we decentralize it? Who has control over that data? And I think some of the biggest challenges that our customers and the market faces today is around just first and foremost, scalability, like, can I get access to that data? You know, second biggest one is security. And last one is just, you know, compliance and governance, like who has access to data, who doesn't have access to data? And as you've seen, there's been a ton of GDPR type problems over the last few years. Uber just got fined 300 million or nearly $300 million for passing data from Europe to the US. That is a problem that they're facing with governance. They don't have the right tooling in place. They shared it. These are some of the challenges that the companies are trying to face. And how you face that, in my opinion, is that I would say the thing that companies should be using is what I would call the modern data stack. Are you familiar with this, Richie? Yeah.

So the modern data stack is a very well known concept, but a lot of different guests seem to define it differently. So I'd love to hear your take on this. What's in the modern data stack?

It's funny because we are, I would say part of the original group of tooling for the modern data stack now almost nine years ago, and there's even like a postmodern data stack and we can talk about that shortly. But what I would define the modern data stack is a cloud based data warehouse or storage layer, generally data warehouse. And you're taking data and loading that into the data warehouse from all the different places that you have it and then you're building tooling on top of the warehouse and you're doing more of an ELT versus ETL type workload. So in other words, the order of operations is different than what it was prior, which is you extract, you load, then you transform and you build this whole ecosystem on top of the data warehouse. And that's in my definition would be the modern data stack.

The idea is you've got lots of data all over the place in different sources. You bring it all into the data warehouse and that's like a single point and then that's the point where you transform things and then you can build something useful. So you mentioned this idea of like ETL versus elt. How does the workflow change there?

Maybe I'll start with just like a history of databases and data warehousing as like a very brief way of thinking. About. So in the very beginning, companies, let's say in the 60s, 70s, 80s, you know, you'd have maybe one database, and that was like your point of record for all the things that the business is running. It's like your sales, your marketing, your accounting, potentially your product. And it's just in one big monolithic database. But the problem then became, okay, well, how do I run reporting on this? And so the problem was like, okay, I could just build a spreadsheet on top of it, or I could do reporting, but they're really slow. So then folks have always wanted to say, like, okay, well, I need to get data out of these systems, or maybe not, and I can just report right on top of it. But if you report right on top of your source system, system, it can sometimes slow it down because these are the types of queries that you're running, are analytical and by nature. So, you know, like, relational databases are not indexed and not performant for doing aggregations. Basically. Basically there's this wish, like, okay, I wish I had a database that could just store everything that could be really fast for analytical query. This is probably like early 90s, right? But there was no such thing. And so what a lot of companies did is they would just take another relational database and they would load some, some of the data from their monolithic system into that other system, and they would do what I would call etl, which is they'd extract the data out, they'd transform it, doing those aggregations, and then they'd load the aggregated data into that database that made it performant because it was aggregated. But, you know, you define maybe 10 metrics, and maybe the business is like, hey, I need an 11 11th metric. And it's like, oh, I'm sorry, we don't have that. We didn't store that data. Like, you got to go. You had to go back and, like, take it out. And so it was like, very cumbersome to actually get a lot of insights. And so there was always this wish of, like, man, I wish I could just load all the data and not aggregate this data. But that sort of ETL workload has survived, you know, since the early 90s, probably even earlier. And I think in the maybe 2007, 2006 timeframe, then came Hadoop. This like, all right, wow, now we have this place, we can just dump everything. And so like, okay, let's take all the data, let's dump it into Hadoop. And Google's doing this, so this obviously must be the right thing. And so everyone just sort of dumped it in. It's more of like a data swamp where you just have like tons of data, there's a lot of duplicate data. You're doing snapshots every day and just like dumping it in. And then you spend a ton of time on top of that trying to make sense of that data. Right. And I would say this is the first step towards the modern data stack. But the problem is because the data is not organized, you end up spending a tremendous amount of time, energy and effort just essentially ETL ing the data once it's been loaded there. So that's like 2000, 6007 and then eventually 2013, 2012 ish redshift comes out. So Redshift is the first cloud based data warehouse. And actually I skipped a little bit of set the. There were a few other on premise cloud on premise data warehouses that are column store data warehouses that are designed for analytical querying. So the way in which that they're actually the data stored is designed for doing this analytical querying. And so there were some of these, like the TZ Vertica, things like that. So that was kind of like what everyone focused on in the early 2000s. But the problem with them is that they were constrained by storage and compute because it's an on premise box, it's in your basement and you're like, okay, I can only drive as much as possible off of the box that's in my basement. Similar to like on your computer. And so once you've loaded up with too much data or you're trying to do too many queries against it at the same time, you end up with like really slow queries. Then people are like, oh, hey, I'm waiting like forever to get my data out of this system. And everyone said, hey, I want these column store databases, but I want them in the cloud, right? So that I can, I can scale them out horizontally. That was like the holy grail. So that's where eventually Redshift takes an on premise database and makes it available in the cloud. All of a sudden everyone's like, oh my gosh, this is the future. This is where, you know, this is where we're going. And the really great part about that was that it somewhat democratized the ability to use these column store databases. Because in the past, the Teaser and Vertica and all these other data warehouses were so expensive that most companies couldn't actually afford them. And so now you have this massive rush towards, hey, let's just move everything in. And the workflow then became take the data, load it into the warehouse, and then do your analytical querying on top of it. And, you know, but when you load into the data warehouse, make sure it's organized appropriately. And the thing that this then allowed for businesses to do is to automate this first step. And in the previous world, the types of aggregations they were doing was always custom to your company, so you couldn't automate any of it. So you had to build out your own ETL from scratch all the time, every time. And so now you have this like automatic replication of getting data into the warehouse, and then from there you build the transformations that you run on top of that. The data modeling is very unique to your company, but instead of spending your like, engineering time building these pipelines, you could basically automate that piece and spend engineering time just doing the actual, like, analysis within the warehouse. And so that's where fivetran came in. Fivetrans is like the replication product, loading data directly from all the different places that you have it into your redshift cluster. And that's what I would say is the modern data stack. And then on top of that, there were more modern BI tools like Looker who became available that were running directly on top of the database. Versus in the previous world it was like you ship some data to your computer and then you'd run these local queries around it, more like a spreadsheet. And so those are some of the things that are happening. I think the other big trend that we saw over that time frame was that you went from these monolithic databases where you had like one database or one SAP system or one ERP system, to having thousands. Because as soon as the cloud took off, then it was like, all right, great, now there's like a million different services to solve every problem that you have. And so companies went from like maybe one to five data sources to like maybe in the 2000s, like let's call it five to ten. And then in the 2000s, it's like 100 to like a thousand now or even more. We have customers who have like 10,000 different sources and databases that they're running or more. And so like, that problem becomes even more valuable to solve in an automated way to get all that data into a centralized place. So that's like the brief history. And then there's the Neo age of this, which I guess I might as well, since I'm talking about it, if you don't mind, I'm going to keep going a little bit, which is what happened in 2015 is that Snowflake became available. And Snowflake is really the first or big career. But Snowflake particularly was the first commercial data warehouse that was designed for the cloud. In other words, it was not constrained by compute and storage. Most databases in the past were always like, you have a box and that's how much storage and compute you have and you're constrained by that. And Snowflake was like, no, no, no, going to use S3 for our storage layer and we're going to use EC2 for our compute engine, but we're going to scale it out infinitely. Like you want to spend more money, we will let you spend until the cows come home and you're going to have the fastest queries ever, but you're going to pay us a lot of money for it. And so that was really truly the first step towards. All right, now I have a fully elastic, very automated cloud data platform or data warehouse that I can just load everything into. And that was like a huge change. And I think that really got everyone that's maybe 100 times better than the on premise databases. And so everyone was like, wow, we got to go do this, right? Fast forward to today and the big thing that's happening now is that a lot of these companies are saying, look, I just want to store data in my own S3, but then I want to query it inside of Snowflake or I want to query it inside of Databricks or I want to query it inside of like Redshift or Trino or wherever. And so I think that has been the next wave of this is essentially saying, okay, let's store it all in a repository that's my own repository because all these services like Snowflake are using S3 anyways, and then allow me to query it in different places. And what this does is it gives the benefit of having a future proof commodity place. You can put all of your data that you can also then use in different places so you don't quite get the lock in that you have in previous where you sort of get stuck to one vendor. Yes, you may be stuck to a certain cloud, but you then can use various different compute engines on top of that, depending on what the workload is. So it might be an AI type workload, so you want to use something different or it might be like a classic BI type workload, so you just want to use Snowflake. Right. And so that gives a lot more optionality to enterprises as they move forward. And that is like the really Big shift. And so what, you know, how 5chan fits into this new paradigm? Like, I think we were fundamental to, in helping the shift towards automation into Redshift and Snowflake in the initial modern data stack. And now we've built a connector for our customers to load data into S3 or into a data lake in a ACID compliant file format using Iceberg and using Delta. But the difference between the sort of data lakes of the 2007 and the data lakes of 2024 is that the data is not a swamp. It's highly organized. You can do history on it. You know, you get all the upside of a database, but you also get the upside of a data lake. And that's the, like, holy grail of what people have been wanting since the early 90s. And so that's a really exciting thing that we're seeing right now. It's still early in that second wave of like, postmodern data stack. But anyway, so I just went on for a whole lot of time there, Richie, but hopefully that gives some context to the conversation today.

All right, so it sounds like the dream then is you can scale essentially infinitely. You decide where you're storing your data and where you're storing or where you're doing your compute. And then in theory, the pipelines to get from where your data's stored to where things are being computed, that's easy to figure out. And so how do you get to that dream then? So you said the technology is kind of there, I guess. What processes do you need to put in place in order to realize this dream?

In talking about it, in the way in which We5chan has worked to solve this problem is extraordinarily simple. Like, you want to get data from 5,000 places into a single location, whether that's a data warehouse or a data lake. We can do that for you. You basically go and you set it up, all these different connections. To put it in perspective, setting up a Salesforce connection takes, I think, like five clicks, right? You authenticate your Salesforce, you point it at what data warehouse you want. We go and grab all the data from Salesforce, we create the tables loaded into the data warehouse for you, and then we keep it up to date, incrementally syncing that data every minute or whatever. And if there are schema changes that happen on the source, we persist these through to the warehouse, and basically we take care of everything. So you just end up with like a mirror copy of your data within the warehouse. That makes it extraordinarily easy to access and move all this data. But then there are some more challenging things around compliance and security and all those things that I mentioned. And so this is a problem that every company is facing. Who's allowed to access what, how do I make sure that I don't send GDPR specific data to the US and things like that. And so FiveTra helps very deeply with each piece of this in a slightly different way with the security piece. I mean we have the security sort of like table sticks, right? You have to have the best security, we have the best protocols for doing the connections. Everything is encrypted end to end. We now have a product called Hybrid Deployment so you can deploy all of fivetran behind a customer, your firewall. So like the data processing all happens within your region and your cloud, so data never touches fivetran servers. Like we have all the different options for helping our customers in whatever way they need to keep the data secure. On the compliance side, we use like role based access control to be able to assign all the way down to the table level and the connection level. So you can say, hey, like I want to give this group access only to marketing connectors, I want to give this group only access to the ERP system, I want to give this group only access to the sales systems or whatever. And so, and then that you apply it like at the table level, then you can apply it at the data warehouse level and then you can apply it like even in the embedding level. So like you can really get fine grained about that. And we don't necessarily do those controls or create those policies, but we integrate with all the tools that do to make it very easy for our customers who need to have control over all this to do that still in a very automated fashion. Because at scale you really can't do any of this manually.

So it sounds like there's sort of quite a lot of different moving parts there. So you've got to like worry about all the compliance side of things and the security as well as figuring out like which bits of data need to connect to where and things like that. So I'd love to go through this sort of maybe slowly and just like figure out like where you get started. So let's think of like a common problem. So like for example, almost every organization, they're complaining, okay, we've got some data stuck in silos. The right people don't know how to access it or what's sort of step one in solving this problem.

I think the first step is figure out a use Case within the organization that you are trying to solve specifically around data. Hey, what questions do you have? What product are you trying to build? And then figure out, okay, what is the data that we need access to? Then you go and grab that data. You'll need to select a data warehouse or a data or a data lake and then ideally use fivetrans. You don't necessarily have to use 5chan. You can build this yourself, but there's always a build buy trade off and frankly you'll end up getting faster results through a product like 5 chance and more accurate results and more security and all the things that I talked about you basically I'd say pick a use case. So say you want to understand your sales velocity and you can't really get great understanding of sales velocity because you don't necessarily have everything within Salesforce. Like you want to understand the velocity all the way from the top of the funnel to the bottom of the funnel. And so you have to combine your marketing data from say Marketo to Salesforce data. And in order to do this, you need both of those systems. So I would start by setting up a sync for both of those sources into the warehouse and then starting to solve one or two questions. Hey, let's build a model on top of that that helps us understand the velocity from the MQL all the way to pose one.

Yeah, so I do like the idea of starting with use cases rather than maybe just like going saying, well, we've got all these different data sources, let's just hook them ap but once. So I guess there can be some data sources where it's like if no one cares about it, then there's not even any point in trying to hook them up.

Exactly.

Okay, yeah, so I like that. So yeah, the sales example, you've got your salesforce data, Marketo data, whatever other tools you using. And then around that use case, those are the things you need to sort out. Now there've been a few sort of ideas that have been fairly well hyped recently. So for solving this idea. So things like data fabrics, data meshes, do you want to talk me through like what these things are and when.

They might be useful or not in that whole history? Now, like now I've given the background and I can explain this a bit more. There's been this concept of federated data. And so federated data is like the data fabric is. It's this concept where say you have hundreds of different data sources. Instead of taking all that data and centralizing it, you query in real Time, So you'd say, hey, like I need this Salesforce object. This is the exact same use case. Hey, I need these two Salesforce objects and I need this one Marketo object. And I'm going to just query that in real time, get that data. And then usually in the virtual layer you will combine that data and you get your answer. It's like, this is amazing idea because you're like, hey, I could just get all the data I need at any given time and I can just go get it. It's like this holy grail promise though that doesn't usually end up paying off because getting data out of these systems takes so much time. So Marketo, for example, can take up to like a few months to get your initial data set out because of the API limits that they impose on you. And so you're like, hey, I'm going to write this query, ask for it and like two months later I'll get the results. So that's where the whole concept of like getting data out of each system is more valuable to put into a data data warehouse. So you're not like beholden to the source systems. You get all the data out and now you can run at whatever speed and whatever you need. And so that is like the initial federated thing now. Now there's like a more. The fabrics that are coming out now are kind of a version of. We'll take some of the data and load it in different clouds and then you can kind of join it together virtualized. But doing the joins actually you can't actually get the results that you want. You really can't join natural full data together. So like, we still strongly advocate for getting all of your data into a single location. The concept of a data mesh is that each individual team owns their data stack and then you sort of share that data around. So you'd say, hey, your marketing team has its own data stack and it builds its own thing and you have your finance team does their own thing and everyone has their own things. And I think there's always a trade off between centralization and decentralization. Decentralization, you're going to get faster results, you're going to get probably a little bit more customer happiness within the whatever organization that you have. But you're trading off security challenges, you're trading off like most recently there is an example where Morgan Stanley had a huge lawsuit because they had a bunch of decentralized data on computers and then they sold these computers because they're like brand of life thing is, but they didn't clear the data off of it. Right. And so all the data was like sold. And so like that is hugely problematic. And that's because you have all these decentralized owners of the data. And so like, we generally believe you should get all the data together and you can still have different owners and different philosophies around modeling, but you can put them all together within the same data center. Right. And I think there's a lot of benefits than being able to join that data together. So like, again, if you haven't figured out, data centralization is like the approach which we still highly advocate and fivetran is that's obviously how we focus on moving data for customers. But we can support meshes, we can support other ones. If you decide, I want to have fivetrade load to redshift for one team and Snowflake for another team or a data lake over here for other team, we can do all of those things. It's just not what we advocate as the best practice.

Okay, so it sounds like centralization you mentioned is better for security. So anything that's like highly sensitive, you probably want less centralized. And then maybe decentralization gives you sort of speed advantages for like less important data set, Something like that.

Yeah, Like, I mean it's more speed and workflow, right? Because imagine that you're just like the marketing team and you have a question about marketing thing you want to go get. You're like, oh, I can just like tap the guy next to me who like is the marketing analyst and I'm going to ask him for this thing and he's going to go and just query the data quickly and give me, give me the answer for it, right? But the question then becomes like one, is this person even supposed to have access to this, like central IT who's responsible for understanding access controls has no idea what's happening over here. So security is like challenging. And then two, just quality. Like how do you know that it's gone through the quality in QA that it's actually correct? Right. And so there's some like trade off. You want to have like your fast and dirty stuff that you're like, hey, this is directly the answer that I want or I'm getting. But your hardcore dashboards should go through probably more of a centralized process that you are evaluating. Like, hey, one, are the metrics correct? Are they governed? Two, is the sensitive data, who's it being accessed by, Et cetera, et cetera. Right. And so when you centralize it, you take all this Data and your central IT team and your security team can understand what data is getting moved. You can use fivetran for example to say hey, I want to block PII data or I want to block phi data. I don't really want to certain tables to load from this highly secure thing into the centralized location because it's really important. Or hey, I actually I want to load up. Like what a lot of people do is they use the medallion system where like you take all the data you put into a raw layer that is highly secure, you know, that's the bronze layer and then you go to your curated silver layer that your IT team and everyone else has combed through and you have good governance on it and you're like great, okay, now other people can go and access this and can build to the gold layer on top of that, but they're not accessing like our critical sensitive data and they can apply the role based access controls there.

I like that sort of easy color coded system for like how important is this? How high risk is it if something goes wrong here? I'm curious as to. Does what you do with your data affect the org chart? Do you need to change your team structures depending on your data strategy?

I think org structure and strategy do need to go hand in hand. For example, like if you invest heavily in a decentralized set of teams, teams and you want to have a centralized strategy, you're at odds with that, right? Because you're never really going to have the right team structure and organization to support a centralized team. Now what we sort of advocate for is that you have a large and strong centralized team who's helping to both bring all the data in, curate do those different levels. But then you end up having still like small analyst teams that are decentralized. And what this allows for is you have someone who sort of speaks the language of data. So say in that organization who can communicate back and forth and say hey, these are the key metrics that I wanted. These metrics are not quite defined, right? Or whatever. I'm going to run a like really quick and dirty thing here. And so you get this like fast response but then also someone that can communicate back to the central team. So that's what we've seen success with. But again like every company's different, the size of every business and their sophistication is sort of different. So it's like you have to sort of think through those.

Am I right in thinking then it's going to be the data engineers who are more centralized and Then the analysts are kind of a bit more spread out and closer to the commercial teams.

That's what we've seen success with, frankly. But you know, when you get to really big sizes, you end up having multiple central data teams that focus on different areas. And so like again, there's always a decentralization centralization challenge. You go back and forth over time depending on the size of the business. But for us, we're about a 1250 employee company right now. We find that centralization is a really good path forward for the future to come. And what I would imagine is that we'll continue to have a centralized data engineering team for many, many years, but the analyst teams in each of the groups will probably start to grow over time just to help support each of the different functions directly.

You mentioned the idea of having sort of quick and dirty analysis where you can just ask your colleague for some results. I guess another one of these sort of holy grails of analytics is having self service capabilities where anyone can ask a question about the company data and be able to get their own answ. How do you get closer to that?

I think this is the holy grail, right? This is the democratization of data. Like anyone can ask any question that I think is the big promise. The challenge that I've seen in practice is even when you implement this, people don't always know what to ask. They don't know what questions they're supposed to be asking. So there's sort of one side which is the human element of it. And I think AI is going to play a big role in this. And then there's like the infrastructure of. And so from our perspective, having a centralized location for all of your data then puts it at a place where you can have people ask questions of all of the data. So if you have this decentralized setup where you're like, hey, marketing can only ask about marketing, sales can only ask about sales. Well, what happens when you want them to think about these other things? Like sales might have a really interesting idea about this or hey, product might have a really interesting idea about what's happening in marketing. So that's why like if you have it all centralized, everyone is able to ask questions against it. From an infrastructure perspective, what we see oftentimes is there's another piece to this which is back to that centralized and decentralized state. There are times when the central IT team says, hey, I want to make sure that we have control over the data that's moving in and out. But if the central IT team is actually responsible for always setting up every pipeline or moving that data. They become a huge bottleneck. So what they want is this thing where I can control what types of sources and what you're allowed to move and I can see what data is being moved, but I'm going to give control to the various groups to do this. And so fivetran actually has, we built a tool around this, we call it Connect cards or powered by fivetran. And what it allows is the central IT team owns the fivetran account, but then they can give smaller accounts to each of the different teams. And since it's so easy to set up, they can say hey marketing, you're allowed to only load marketing connectors, but you can add whatever you want within the marketing realm. And then they just go and add as much as they want. It automatically flows through central IT to the central location, but maybe gets put into a schema that they have access to and then shows up immediately and then they can start querying it. You know, you have to think through that. But now it's possible where it wasn't before to get this sort of self service aspect to data in a centralized way, which is very exciting. On the other side like just training people how to ask the right questions, it's hard. You need to go into ask really promoting education within the organization around what questions are the right questions and what are good analytics. Frankly like we spent a lot of time in stats. Our CEO George is a big statistician type person. He spends a lot of time educating folks around like hey, like when we're thinking about experimentation or other things like you really need to be thoughtful about do we have enough, enough data to actually. Or enough like and to actually look at the results here with stat significance, things like that. And so that just comes back to education. I think the advent of AI and the rays of AI is going to be the thing that makes this easier for your average employee within the organization.

I like this idea that well, teaching people how to ask good questions is maybe the hardest thing. And so you do need some level of education around the company. I'd like to get into that a little bit more. So in terms of skills that people need in order to be able to ask good questions. What's like the one thing that you would try and train everyone on?

So I was at an event last year where I ran a roundtable with a bunch of CIOs and there were all different levels of companies of CIOs and all different levels of I would say, data readiness across those companies. And some folks were trying to figure out, like, how do I build the modern data stack? How do I get through each of these pieces? And some were like, I built it years ago, we're good to go. And the thing that the folks who had already built this, the advice that they had to the folks that had not really built this and how they changed the culture was what they found is a lot of times the central IT team say they're responsible for building analytics. They would go to each of the departments and say, hey, what data do you want? What metrics do you want? And a lot of times the response was, well, what data do you have? It was like, the teams hadn't really spent any time thinking about data. And so they don't want to be stupid, but they were like, well, just give me everything, right? And the way in which the central IT teams found success in doing this is you go and say, hey, how do you know if you're being successful? Which ways do you know? Like, I just was successful. And teams are very easy to answer that, hey, like if I hit this particular thing or our revenue grows to this amount or this particular activity. And so you start with very basic ways of trying to think about measurement and you figure out the metrics around that and then you start to like, build that in, hey, let's start with those few first metrics, let's get those delivered, let's start looking at those. And then you just continue to grow from there. And I think there's a lot of books around this. And so I don't know if that gives back onto the question that you asked.

No, that's absolutely brilliant because I think cross team communication is definitely one of the hardest things, especially when you got commercial teams speaking to data teams and being able to say, well, what constitutes success for you? That's a great sort of object line in terms of like trying to get to what data do you need? And going right back to the start of this, you were talking about how the tooling landscape, particularly for data engineering, is just evolving incredibly quickly. So with everything changing, what do you think are the most important skills for data engineers right now?

I would say that just learning about the new tooling is like the most important thing. I think there are certainly new tools that are widely accepted today. The modern data stack. I would say, hey, if you don't know about the modern data stack, you've never used it. I would highly suggest going and spending some time learning it. There's a tremendous amount of material online that you can go and learn about it on YouTube or otherwise. And so that's the first one. I think data lakes, in my mind, are the future as well. And so we've obviously invested a tremendous amount of energy in this. I would say we're one of the only vendors that provides flawless access to data within a data warehouse that is highly organized. And, you know, I'd say spend time learning about the modern data lakes on top of that, I think trying various different modern data stack tools on top of it and just understanding what all is out there, because there's so many now. Like, if you think about the modern data stack as one big combination of tools that all work well together that create this solution for the customer, it's now like thousands of little tools that are doing very little pieces here and there. And so there's a lot to learn. And at some point, I'm sure there'll be more of a consolidation. But just understanding what do each of these things solve, is it worthwhile? Is it not? I think a lot of engineers oftentimes think as well, hey, I should just build this whole thing. Why would I use any tools for any of this? And that's the constant build versus buy. And I'm not suggesting that you should buy everything, but I think that there are a lot of tools in the modern data stack that engineers sort of poo poo because they say, hey, I just want to build this one myself. I want to build this all myself. And what I think is that just like Stripe or other types of infrastructure tools, you can build and have tooling at the same time. And so I'd say fivetran is similar to that. If you want to build a data application and you're a data engineer, instead of spending all your time building these connections, hey, like, you should probably just leverage a tool like fivetran. You can do all of it through API. You can build it all underneath the hood, and just like Stripe, it can just run all your connections for you for your application. Or like, one of the areas that I've seen success or sort of interesting is a lot of applications that were built in the 2000s. Even cloud applications had to build a reporting layer into them. You know, imagine you're building like a simple application and you spend half of your time building this complicated reporting layer that's not very good. Well, that's like half your R and D effort that's just being spent on like, building out something that people are maybe going to use or maybe not. And so what the trend is now is like, just don't build a reporting layer, just load this, send the data to fivetran and that way the customer can build their own data layer. But you as an organization can spend more time on your R and D effort building the application yourself. And that's going to make you much more competitive and faster for building. So I mean, hopefully that gives the readers or the listeners a bit more direction.

Okay.

Yeah.

So if you want to work in data engineering, probably don't spend too much time worrying about the reporting side of things. Just focus on like, I guess, the data movement side of things and then data warehousing or data lakes. And that's, I guess that seems like the place to start. But you also mentioned like the postmodern data stack. So do you need the postmodern data stack tools as well as the modern data stack tools?

They generally sort of play together. So you could just use a data warehouse, a cloud data warehouse like Snowflake or Databricks and build on top of that. Or you could, if you add a data lake, you're adding it underneath it in the stack. So you could still use S3 and then Snowflake and then build on top of that. Or you could do a data lake and then not use Snowflake and just put your tooling directly on top of it. And so I don't think that it necessarily changes the stack that much. It just gives more optionality to using additional tooling and gives more ownership of the future of where your data resides and how you're going to leverage it. Right. So when new tools come out, you're like, oh great, I can just throw this right on top of here and I'm good to go.

I guess you're not going to run out of tools to learn no tools.

Today, especially with all the advent of AI. And I think just as like a super quick comment on AI, the really cool thing about AI and the thing that we're hearing so much right now, in my mind, it opens up the ability to do aggregations across your text data or your video data or whatever. Whereas before it's like, hey, I want to count the number of humans that signed up last month. It's like, okay, well I have to have that count somewhere and it's going to aggregate and whatever. It's like, hey, I want to know the number of people whose favorite color is blue and it just like can grab that. Right. Things like that that are much harder to get because they're deep within the actual text data and so that is going to open up a whole lot more information. And that's something I'm super excited about. And we're seeing a massive uplift of our customers and new prospects who are using FiveTrain to get all this data together. And generally the first step in building an AI strategy is building a really solid data strategy. Back to that whole, the CIOs that I met with last year, the ones that were really advanced in their AI strategy were very advanced in their data strategy. They had a centralized strategy, they had governance put in place, they had replication fully in place, they had modeling in place. And then adding AI on top of that is actually quite easy because you're just using the same infrastructure that you already have. I think the mistake that a lot of companies fall into is that they try and build a separate AI strategy with a whole separate set of tools. And what you end up realizing is you just build it all from scratch and you spend all of your time doing the infrastructure rather than just using all the tools that are already available and then building on top of that. And fivetran is like a core part of that infrastructure. And that's why I think we're seeing this big uplift in success with all of our customers who are using us for AI. Like sex with that Fifth Avenue.

Okay, so I agree with you that it's important to have good, well governed data in order to be successful with AI. Does that mean that your data strategy needs to come first and then your AI strategy follows it? Or is it the other way around?

I just think that you should think about them as one strategy, right? Which is you're getting all the data from all these different places. If you take all this data and you centralize it, then you have it all in one place. And you can use some of this data for bi. You can use some of this data for AI. You may use some of the same or overlapping data and you may use different tools. So like again, the data lake strategy and using a company like 5chan to load all your data into your own S3 then gives you the freedom to say, I'm going to use Snowflake for this set. I'm going to use databricks for this set. I'm just going to build my own RAG application here on top of the data that is conjoined between the two of these. Right? And that freedom is really important. But I think again, like if you have, like what we've seen is sometimes people will say, I'm going to take all my data for my AI strategy and load it over here and then all my data for my BI strategy and load over here. Then you sort of end up with different answers because you've built it in different ways, different teams, how the data is replicated. You sort of lose this whole governance component as well. And then you have to rebuild all the same things, the same governance, the same control, the same, like, security controls, the same quality levels. Why not just do that once versus having to do that twice?

Okay, yeah. Doing things once rather than twice does sound like a very appealing idea. And I think because generative AI has been incredibly hyped over the last few years, there's been a lot of executives and boards kind of pushing for more generative AI everywhere. Does that take away from sort of funding and data initiatives? Is there a competition between data and AI, do you think?

There is, unfortunately. Last year there was a large company we were working with in Europe that said, they went to the board and said, hey, I need funds to build a data platform. And they said, no, there's no money for data, but you can have unlimited funds for AI. And so this person was like, okay. So they went back, revised their plan, came back and said, hey, I need an AI platform. And that was basically just building a data platform. And they just changed the language and they're like, yep, we're good to go. And so I think, like, it's more confusion. Like, I think there's a lot of folks in the market who don't necessarily understand what a good AI platform looks like and what a good data platform looks like, and the fact that they're sort of the same. And I think there's this, like, craze to be like, let's spend a ton of money on AI. Let's just go build it. It doesn't matter. Let's go crazy. So you end up with really weird things. Like, we've worked with a few vendors because we're doing a lot of stuff internally on our own AI, and we like to always understand what folks are doing. And so we built our own internal chat, but we talked to a few vendors along the way, and we're like, hey, can you build a chat for us? And they said, yes, absolutely. Step one, take all your data, put it in a PDF and send it to us. We were like, wait a second, you're taking, like, snapshots of data over time, and that's what you're using for your live chat app versus like, why don't you just use FiveTrans, load all the data in have it updated every minute, and then your data is like, live to that level. And so I think the whole world is just not really caught up on the AI side for what has happened in the data side over the last, like 10 years. Like, that will happen quickly and it is happening. But I think a lot of education is just needed at this moment.

Absolutely. That's an absolutely fascinating story. Like a genius hack, just taking your proposal for data platforms, find or replace data with AI and then suddenly it gets funded. I'm sure you can do that with a lot of things as well. It could be, take anything, replace it, put AI in there, and you'll get more money.

True, true. For sure. I mean, AI is, I think it is sort of coming into all the different areas of the business, so you could kind of do that, but especially with your data infrastructure. That is the key to having a good AI story.

I see. Do you have any other stories of AI insanity?

I mean, no crazy ones that I think I can speak about. One interesting one that we've done that is unique, or when I was going back to how do we help the broader mass of people within the organization work with data? And I said AI is one of those areas that I think will actually help a tremendous amount. When we were building all these different connections, we have over 500 automated connections today and we're adding something like 200 a year. Well, part of that is like, we have to go build, like we actually have to build the integration into an API. So we have to go figure out what the API is. We have to have engineering figure out like all these pieces. We wanted to figure out how can we scale this. So we built a machine learning module that you could point at the API. It grabs the docs based on the docs, creates the actual integration in a copilot setup, and then we have a human jump in and look at it. But instead of Engineer, we have more of an analyst. Right, because the analyst is the one who actually knows what the data model should be. They know what more context around what the application does. And then very easily can say, oh, no, we got to tweak this, tweak that, tweak this. We're good to go. And that's made our ability to create new connections significantly faster, like from weeks to hours. And so I think that is not exactly like analyzing data, but it's working with data. And I think this is going to happen all over the organization. That's one way in which we're heavily leaning in to make sure we can Leverage this as well and support every single connection across the planet for our customers.

Okay. So again, yeah, it's scaling stuff so every bit of data gets connected somehow. I like that.

Right. There's like thousands. I mean, think about it. There's probably like 10,000 different applications that across the world, probably even more. Probably 50,000 startups all over the place who have some sort of important data for our customers. And so FiveTrans is the single platform for enterprise, as I said, enterprises large and small for centralizing data from everywhere. And so you have to do things like this to be able to access all those. Right. And build all those connections or one of the other things, like we're coming out with a product later this year that we're very excited about for our customers to leverage a lot of the core technology within fivetran, but build their own connectors. So imagine it's like probably 25% or 10% of the work of someone just building it themselves because they can use all of our centralized functions, but they have access to build whatever they need. So now they can go and add these very quickly. It's another thing that just helps our customers.

I like it. Yeah, I have to say I do like this idea of just like basically pointing and clicking and creating a pipeline. It's like not having to write vast amounts of code, just stuff just working. It's very appealing. So, so just to wrap up, what are you most excited about in the world of data engineering right now?

You know, I have to say that again, this is sort of, it's something that I'm excited about from, that's coming out from fivetran. And so it's a bit of a, you know, a plug. But in about a week and a half we are launching a feature called Hybrid deployment which takes our fully managed cloud based tooling and makes it available behind the customer's firewall for replicating. And while this, this sounds like, oh, wow, that makes sense, you know, it's very easy. It's pretty complicated in the sense it's very easy for our customers. They get this cloud experience because it's still controlled from the cloud. They still go to fivetrain.com and they can access whatever, but the actual processing happens behind their firewall. And so to make that really seamless and very easy, it took a tremendous amount of work. And I think this is really the future for the industry in terms of the architecture for security purposes. And I'm, you know, I'm very excited that we're on the cutting edge of it.

Yeah, so I like the idea of, like, good experience for customers in the cloud, but also it's in all these security stuff. And it's just kind of. It's just an engineering problem. My engineers say things like that, but yeah, I like that. Do that. Do the hard work and then the customers don't have to.

I will say the last thing is we tried to make our product so easy. Like, literally when we first launched it in 2015, I had my mom go through the workflow to set it up because I want it to be like that easy for customers to set up. And the cool part is like, now it's that easy through the UI or it's that easy for engineers building on top of 5chan, where they can go and very quickly get set up and they can run and set up, like all their connectors. Like, you know, some of our larger customers, like Rappi, which is, you know, sort of like an Amazon of South America, there's thousands of different databases that they're running through this system. And it's all set up programmatically. Right. Hey, whenever I set up a new microsystem, microservice within a new region, spin up fivetran, run all the data for it, get it centralized, and so it just becomes part of their engineering stack at the end of the day, which, you know, I think hopefully for this, for this audience, should be quite intriguing.

Any final advice for organizations that want to improve their data engineering capabilities?

Modern data stack, Postmodern data stack, I would say learn about those, figure out a strategy for moving pieces over to it. And I would urge people to listen to your podcast so they are learning all the new things that are coming out from all the interesting folks and use five chain. We make it extremely easy.

All right. I do like the idea that listening to databramed is the solution to everyone's problems. Excellent. Yeah, and I love all the different names for the data stacks. Your modern postmodern, they should be like an art deco data stack. That would be wonderful.

Postmodern is something I sort of threw out. People have talked about it a bit. Like, I. It's more like, I'd say the modern data lake is more what I've heard or we've coined, but again, like, who knows? It's like everyone talks about things differently.

All right, wonderful. Thank you so much for your time.

Taylor, Richie, you as well.