#282 Navigating the Challenges of Product Integrations with Gil Feig, Co-Founder and CTO of Merge

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

Integrations are a full company strategy. You know this isn't this isn't some Post a notification on Slack when something happens, we're talking what people listening to this podcast are familiar with millions or potentially billions of rows of data moving daily. What I think is really cool is we are the picks and shovels for all the new gen AI companies. We firmly believe that integrations are actually sticky in both directions. It's the same thing with integrations do not build 50 connections. It just makes no sense.

Welcome to Data Framed. This is Richie the number of pieces of software that every company uses has skyrocketed over the last few years. It's tricky to keep up with learning about all these tools, but there's another big problem this causes. The number of connections between pieces of software grows faster than linearly. It's skyrocketing squared. Trying to manage how applications talk to each other puts a huge strain on software engineers and data engineers, particularly when it comes to core business software for managing payments or HR systems. It seems like there's a lot of scope for automating grunt work for big productivity gains. So let's find out how to do that. Our guest is Gil Fye, the Chief Technical Officer and co founder at Merge, a unified API platform. Previously, Gil was the head of engineering at Recruitment Platform Untapped, building on a background as a software engineer at Wealthfront and LinkedIn. Bullet Siri's advice on managing the space between the software. Hi Gil, welcome to the show.

Thanks for having me, Richie. Happy to be here.

Wonderful. So, just to begin with, tell me, what is a product integration?

Product integration is essentially just a data connection between two pieces of software. So it's any means for transferring data between that software, whether that's pushing, pulling data. It's often thought of in terms of APIs these days, but historically it goes way back. You have sftp, you have file transfers, you have other means of transferring data. Though again, these days, generally it means a native API based integration between two SaaS providers.

So integration sounds like a good idea, but maybe you can spell it out. So why would you want these things? What are the business use cases?

Yeah, I can think of a ton of business use cases. Happy to give one that, you know, Merge helps power, but it's just a generic use case that a lot of companies can use. So Merge often sells into spend management and you can think of that as essentially business credit cards. So Amex business, but competitors, you have Brex, you have Ramp, you know you have Airbase, there's so many of these. Navon has their own now and essentially what they want to do is make it so that these credit cards can be automatically sent out to all the employees of company. It's beneficial for the companies because they want to say, hey, give my engineers $20 a day for lunch. But it's really hard for them to go into Brex every time someone joins. Add a new employee and then if they get terminated, revoke that access. But also make sure the rules are set up correctly. The manager approvals are set up. It's really complex. But all of that data is contained in a company's HR system who needs to approve what their job title is. So rules can be set. So instead of actually having someone manually go at all this information to Ramp and Brex, instead they now offer native integrations with the HR systems of customer platforms. The problem being every company uses a different HR system. Workday adp, you have rippling justworks, there's so many. And when you want to launch something like this, you have to integrate with all of them. That could be 50, 100 integrations. And so essentially it has become just a really big problem and a really unmanageable web of connections that companies are having to manage these days?

Okay, that certainly makes sense. I mean I think about how expenses worked like a decade ago and it's like, well, you weren't sure whether you were going to get approvals. You had to like drop an email to your manager before you bought something and then wanted to go through is like, well, I take a photo of the receipt and upload it. And all these like platforms managing expenses, very, very cool. Like it does save a lot of boring admin work, but then you need to make sure it hooks up into all your other systems for you to actually be able to take advantage of these sorts of things. So yeah, I can certainly see how that's a very cool use case. So has something changed in the sort of software landscape or the product landscape. That means integrations are more important now than they used to be. Oh yeah.

I think a lot has changed over recent years. So you kind of started to see maybe about five, 10 years ago, the breakdown of these software suites, the work days and the net suites, and they just do so much. And you had all these startups popping up that Specialized and we're like, we're going to do time and attendance, but we're going to do it incredibly well. We're going to build it modern with AI. It's going to be deeply connected to other platforms. And so because of that, these new startups needed to integrate with these bigger tech companies to pull in that data. So they started building integrations with all of them. And then consumers or business buyers saw that, hey, all of our software is starting to integrate with all of our other software. We're starting to see these upstarts that do things really well. We want to make sure it connects to our existing systems. So they're buying things. It was doing it. Then they started demanding actually that their existing systems connected with everything else. They're like, hey, all these startups have integrations with everyone else. Why don't you? And so that's where you kind of saw this just really take off. Customers really demanding integrations is what drove companies to do it. And now you even see the big software suites actually integrating with one another, integrating with their own competitors. And there's a lot of reasons for it. It makes them more sticky. It has a lot of positive effects in their business. And I think in recent years that that became very clear and that's why you're seeing more of a trend even among the companies that were originally really hesitant and locked their data down.

Okay, so I like this is a customer driven approach. And certainly think about the tech stack, at least at Data Gam and I'm sure almost everywhere else, it seems to have grown a lot. Like every year there's more and more different software products to buy. And once you start thinking about how you connect them, then I guess if you've got like end products, it's going to be like N squared connections between all these products or that order of magnitude anyway. So I can certainly see how the number of integrations is going to have dramatically increased. I guess that leads on to like, how do you scale this? Then how do you go from well, I want to build a software integration to I need hundreds or thousands of these things.

So I think everyone ends up going there. But I think the problem is you don't realize it upfront. And I can give you an example from my last company. We were in recruiting and we had to integrate with applicant tracking system. So places you go to leave feedback on candidates. So first we built an integration with Greenhouse. Then we had customers say we use lever integrate with that. So we did, but we never built. Cause we didn't really research. We didn't know it was early, but we never really built a system that was meant to scale to all the integrations our customers were using. And so ultimately we ended up just kind of frankensteining one on top of the next until we hit this point where we were like this has to be all redone. But also we don't have the resources to even add one more integration because we're maintaining them, we're managing them, we're supporting customer requests that are coming in about bugs or issues or even just someone not knowing how to onboard. Like it was just, it was insane and it was killing us. It was killing our business. We were turning down really big deals because we couldn't build those integrations. So ultimately that's really what led to the idea of merge in general was companies should not be building this in house. Companies used to think it was right to build their own data entire warehouses with, with all their own servers and racks. And it was pretty clear you shouldn't do that. Move to AWS. It's the same thing with integrations. Do not build 50 connections. It just makes no sense.

Okay. I like the idea that it's kind of the same as moving to the cloud. It's like you just want someone who else to manage the things that aren't part of your core business.

Exactly. And it's ongoing. It's, you know, a lot of people I think think about these integrations as a one time build it set up, done. Because we're all proactive engineers who write a ton of tests. And then as we see in the data world a lot the data doesn't follow the contracts, the docs are wrong and we end up getting data that should be impossible to get and we have to be able to handle that. So there's a lot of maintenance and management on merges end we have a lot of tooling and you know, we're not built out to manage that. But otherwise that does fall on the business who builds those integrations. And at scale it's just not possible to maintain and manage all of that. Okay.

Yeah. I think the idea of terrible data quality and impossible data actually occurring in real life, that's Going to be something very familiar to a lot of our listeners. So yeah, it's nice for someone else to handle it, I suppose. All right, so talk me through. Who needs to care about this stuff, like who is normally responsible for building product integrations?

So one thing that we like to emphasize at Merge is obviously building is part of it, but integrations are a full company strategy. You know, this isn't some post a notification on Slack when something happens, you know, we're talking what people listening to this podcast are familiar with millions or potentially billions of rows of data moving daily. And so obviously, of course, data engineers are involved and software engineers are involved. Product managers actually tend to be the buyer because the value there is add a ton of integrations at once. They're the ones really working on that data. And software engineers are going to be the sort of evaluators, the ones who get the technical buy in. But you also have partnerships, design, you have technical customer support, you have pre sales and then any other post sales roles involved. You know, imagine you launch 50 integrations and a customer's like, hey, my API key is showing an error when I go to try to generate it. How does your team know how to go handle that for one out of the 50 integrations you just launched? So that's why we say it's a full company effort. But of course data engineers and engineers are responsible for that build.

Okay, so I kind of like that engineers are at the core of this, both data and software. But yeah, obviously you do need all that kind of supporting infrastructure around, particularly on the product side and then support side as well. So maybe some motivation then. Have you come across any success stories where companies have built a load of these integrations and then it's paid off with some sort of business benefit?

Yeah, absolutely. I gave the example of Ramp and Brexit. I think they're a really exciting one. I can give another example here, which is Guru. So Guru essentially provides enterprise search for businesses. So they effectively come in, they log into Guru, they connect all of their business system. So that'd be their file storage system, whether that's Google Drive. Then they would connect their ticketing system, Asana, Jira, those sorts of things. Hr, you have Workday, you have adp, justworks, and then, you know, a bunch of other platforms too. I mentioned ticketing. We also offer accounting CRM. So it continues to go on. Companies connect basically everything they can that all gets indexed into Guru's search. And then they provide AI powered enterprise search for businesses. They need as much data as possible. And they're working within the confines of the best models that are available to everybody. And so having that data is giving them a real competitive advantage there. They've onboarded thousands of customers onto their platform and onto merges integrations, which is really exciting.

Okay, so it just seemed like, basically, if your customer is using, like, these other business software where the integration is required, that's gonna be like, a really important thing just for actually winning the business in the first place. And then also, I guess once you get them set up, then it's gonna make your product a little bit more sticky because, you know, they're then using two bits of software at once. And so, yeah, a little bit more locked in.

We firmly believe that integrations are actually sticky in both directions. So a lot of times, the system that's having mostly data pulled out of it, those companies tend to fear people are extracting everything out of my system. We're losing value. It's not worth it for us to keep this API open. But what they're actually finding is similar to Salesforce, people end up building a real ecosystem around those platforms. So, effectively, the other platforms that are integrating now depend on the data from that source system, and that customer can never get off that source system because their whole business is now running on it.

Okay, so, yeah, definite business benefits there. I'm wondering how it's going to change the engineering side of things. So if you're living in a world where product integrations are easy, how does that change how the engineers work? Maybe we'll start with the data engineers, since that's pretty close to the heart of data framed.

I think when data engineers, when we think about building integrations, the big one is this. I need to ETL a massive amount of data. Let's focus on the retrieving case. There's also writing data back. But let's just say for the ETL case here, you're pulling a ton of data out of the system. You have to build new ETLs for every platform you integrate with. So let's say your PMs. Like, we want to launch HR integrations to facilitate auto, you know, sending credit cards. All right, let's go build 50 ETL. Actually, it's a lot more. It's. It's maybe 10 ETLs per integration. So you're talking 500 ETLs. You're talking just a mass amount of maintenance. All these problems that are coming up, you don't have to deal with any of that with a unified API, because you. You just build one. You Build one per piece of data, and the unified API handles the actual differences between each platform. So that's, I would say, probably one of the biggest ones. The next one is, yeah, you really don't have to deal with the bugs, you don't have to deal with the data cleanup. You don't need platforms like Great Expectations or other, other things to make sure that your data is really in order here you can really just rely on merges, normalization or whatever platform, and then also observability. So Merge, for example, offers a whole suite of observability tools. So these could be things like data connection issues, you know, rate limits. Anything that's popping up is all clearly documented and spelled out. So it's just really easy to figure out what's going on with any integration. No need to build anything to monitor that.

That's kind of interesting. There's three different stages there of the data engineering workflow that's affected. So it's like the original, well, how do I do the ETL stuff? So how do I get the data into my data warehouse? And then how do I check the quality of that? And then how do I observe that it's continuing to run as intended? And all three stages are kind of. Basically, you want those automating mostly anyway, I guess.

Yeah.

All right, nice. And so just related to this one idea that's becoming very popular in the data world is the idea of data products. So the idea that you treat data like a software product and you have like guaranteed uptime and all this sort of stuff and APIs for, like, how it works. This sounds very relevant to the sort of stuff you're working on. Have you thought about data products at all?

In a lot of ways, we are a data product, right? Like our API is just a way of retrieving data for your customers. Uptime, reliability, data quality, all of these things matter. And not just, you know, us saying it. We need our SoC2 and our ISO audits to cover that. And it's not just security, it's accuracy, it's reliability. We get audited for all of that. So yes, it's critically important. And then among our customers too, we're really helping power a lot of these use cases because a lot of them are, you know, things like sales dashboards, pulling all my data from my CRM and display it in a way that gives me real insights over that data. They also have serious requirements around providing a great product when in a lot of ways they're just a wrapper around data. So I think It's a couple approaches. That one's a little bit more of a stretch, but absolutely. This is the ecosystem that we're in.

So I guess beyond the data engineers, like how is it changing the workflows for other people around the company?

So I mentioned post sales before, you know, that's a really, really relevant one. It's anyone who gets involved with the deal after it's closed, but it actually pre sells as well. You need to be able to talk about these integrations, you need to be able to support them. And when something that a data engineer or software engineer works on becomes customer facing, but it's still a little too technical, support can't really handle it. You end up having to rope engineers back in. And when you have engineers directly connected to customer requests, that's the depth of your business. Right. That is so expensive it's impossible to maintain. And so that's where you know these, these observability, tooling being non technical, making it so that someone in customer success can actually handle any incoming issues without looping and engineering is critical to launching successful integration strategy overall.

So my understanding of this is normally if it's pre sales, then your account executive is going to sort of try and handle questions. If it gets too technical, they rope in a sales engineer. And I guess the idea here is that maybe you don't need that expensive sales engineer because their time is always precious. And then I guess on the post sales side it's like normally the customer service manager is going to be helping out until it gets too technical, then you get a solution architect and you can avoid the need for those people being involved as well.

Exactly, exactly. But, but ideally, you know, when you think about these integrations, it's like you want every single one of your customers connecting most of their platforms. And even a solutions architect, like having anyone involved can be expensive. And so productizing the onboarding experience itself is very, very important because you just don't want anyone else involved. And if you do have people involved, that's where you want customer support to be able to provide that frontline support using tooling that engineers built. That's very clear about what's going wrong.

So in terms of trying to implement this, how do you go about it? Is it start feel like you try and build a product integration yourself and see where you're falling over and then, and then try and scale it once you've worked out the kinks or what, what's the flow here?

It's funny Richie, we always say that businesses that have built at least one integration are our best customers because they have felt the horrendous pain that comes with this. I personally love getting on a call with like a. No, no offense to anyone, like a know it all brand new engineer who's like, well I built a Slack integration in college and you know, it sent notifications and it's just not that it's so complicated, it's so much deeper than that.

Actually I don't know whether you can name names like what are the worst sorts of product integrations to build them.

So I mean there are a lot. But data rich integrations with any core system of record is going to be tough. So look at a system like HR when we think about data volume. So hr, you have all the people in a company that's not going to be too high, right? Like even at Boeing, a million people, maybe, you know, 10, I don't know, whatever million if you include terminated. But then for each one of those, for every week that they were there, they got a paycheck that's pretty bad. Then when you look at something like an accounting system, you're talking about, okay, every invoice that's ever been created, every account that they've ever transacted with the data starts to go up a little bit more. But worse, you know, you look at something like a CRM system where not only is every customer they've ever talked to in there and every deal they've ever run in there, but they've also imported massive data sets of just millions of companies from around the world, millions of contacts. They have this auto connected to scrapers and things that are just pulling in so much data. That's where you really see struggle. So core systems of record in general, a lot of data, but systems like CRM file storage tend to have a lot of data. And then lastly, the other thing we're always looking at is how old are these APIs? When you have a really ancient API, it has bad access patterns and there's no way to really pull out the data you need. So you often make, you know, this is a query term, n +1 queries where first you have to query for the overall set of IDs and then for each ID you have to kind of look up the data which is really inefficient. You want to kind of look up everything at once. You run into that problem a lot with API integrations and it actually can make it physically impossible to extract all the data in a reasonable amount of time. So you have to work really hard within those limits and figure out what you can do and productize. Hey, how can I best build this integration given the physical constraints that exist here?

Yeah, I can certainly see how, like your CRM system or something like that, it can get huge and wieldy because there's lots of different types of data in there. It's going to get complex. I'm curious, is most of those things sound like text data? Is it mostly text data you'd be working with, or can you go full on multimedia?

Yeah, full multimedia. So we integrate, like when you think about a recruiting system, an applicant tracking system, every single candidate who's applied for a job, or almost everyone has uploaded a resume, potentially a cv that's all pulled, that's all available. When you think about a file storage system, there's metadata, the file structure, but otherwise it's just pure files. That's actually incredibly complex because of the rate limits on these systems. You're trying to sync the file structure while people are also trying to pull files. So, yeah, it can get very messy there as well.

Okay. Yeah. I suppose, like, at the very least you can have things like headshots of candidates in your. In your HR integration, all that kind of stuff. Yeah, I suppose, like a resume is just like a PDF or whatever. All right, so for people who are interested in building these things, what sort of skills do you need?

So I think just experience ETL ing a massive amount of data is very important. You know, you need to distribute the workloads. You need to make sure that, you know, you're pulling the data efficiently. You're not doing it in a way that would leave gaps in data. But Merge has a lot of guides. Us personally have a lot of guides on how to actually sync that data, how to set it up properly. I would just recommend really doing your research before diving in, because it's easy to think about these and it's easy to test them when you're on some small test instance. But the second you onboard a customer with an amount of data, where if you took the amount of data and divided it by the rate limit, it would take like the old window screens, like 475 years to transfer, that's where you really need to go deeper.

That's interesting. Yeah, certainly I can see how waiting 475 years for your data to transfer, that's not a great customer experience.

There you go.

But it sounds like mostly what you need to know is just being able to understand how to work with APIs, is that kind of the crux of it or there other things as well.

Yeah, I think knowing basics of APIs and understanding the importance of data transfer in terms of both an initial sync, so like a data grab and then ongoing syncs, how you can efficiently only pull change data as opposed to repulling full data sets every time. So basics of like ETL syncing.

And are there sort of product management skills you need here as well or is it basically just engineering skills?

So I think product management skills are important because I mentioned this before, you need to rope in your whole company and so it depends what resources you have available to you. If you're at a larger company, Right. You can be that data engineer who really focuses on the engine build. But if you're somewhere smaller, you do need to think about that full company build. And what happens, what happens when customers start reaching out for support? Does our team know what to say? And that's why I think unified API platforms like Merge, you know, they really provide a lot of that. So data engineers can do what they want to do best and PMs can do what they want to do best and no one has to kind of take on every role at the company.

So I know you've run engineering teams before, so do you have any career advice for aspiring engineers?

I think this advice would probably not come off as, you know, extremely original, but I think it's so important and people tend to brush it off. You have to work hard. It's your time to learn. When your early career you just one, you'll never have as much time to code again in your career and you'll never have as much mentorship available to you. It's really important that you're sucking up all the information you can. And when people say in your first three months of work you'll learn more than your four years of college, it's absolutely true as long as you put in the effort for it. I think in my first month of work I learned more than four years of college. And so I just, you know, work hard, invest really heavily in your own learning. Don't take a job that's paying $10,000 more because you expect that the money's gonna make all the difference when it could be a slower growing company, you know, it doesn't have the same number of coworkers to learn from. All those things are just temporary. And I think taking the learning will make you a lot more than $10,000 a year later in your career.

Yeah, it's a trade off between short term gains and long term gains. And certainly I like the idea that learning throughout your career is incredibly important. I feel like, you know, sort of mid career and still learning a lot. Okay, nice. Actually, do you have any career disaster stories you can tell us about lessons learned? I always get that wrong. Lessons learned.

Okay, so I, I actually, I have a lot. I have a couple lessons learned. I think my biggest lesson learned has to be grass is always greener in tech. I think that every company always looks appealing, you always want to move, and you always see the new hottest thing on TechCrunch or the new hottest trend like AI, which I'm obsessed with too. Right. But I think all that being said, every company has its thing going on on the inside. Every company is going through a lot and I think you really need to optimize for yourself, for your learning and for your growth there.

Yeah, I do like that idea. Just optimize for learning and growth. I mean, since you mentioned generative AI, it seems like we always end up talking about generative AI on the show, like whatever the topic is. And I think we've done quite well in avoiding it so far. Is there a relationship between product integrations and generative AI or is that one stretch too far?

So I think actually integrations and just the data engineering role in general in gen AI, what I think is really cool is we are the picks and shovels for all the new gen AI companies. Which is a reference, if you haven't heard this, back to the gold rush when everyone moved out west and it was the people selling the picks and shovels that made the real money. They were the winners and ultimately that's us. In the AI era, you have Merge as a data provider, you have data engineers who are actually making the data flow. Any individual business product that can give people a competitive advantage. Right now, when again, everyone has access to the same models, makes you a pick and shovel who's really going to succeed.

Okay, that is fascinating. I think there's a lot of people going, well, you know, generative AI is where the money is. It's very exciting stuff, but actually going a few layers down into the data engineering, into the kind of fundamental infrastructure that's like a much more reliable source of value, I suppose.

Yeah, absolutely. And as much as AI continues to change, whatever happens, it's going to need people like all the listeners here and products like Merge.

All right, so what are you most excited about in the world of product integrations?

Well, a lot of things. I think one growth, really excited about it. I'm going to repeat AI Because I think that it's going to also change how the world of product integrations works. It remains to be seen, but right now, building API integrations is incredibly cumbersome. It's complex, it's tough to get the data that you want. And I'm excited to see how AI changes that landscape just to make data exchange a lot easier.

No, I do like the idea of just making things much easier for developers, for engineers, and simplifying a lot of the stuff where it's, I guess, integrating different products. It's a niche idea of a good time. So I guess the less time you can spend doing that and the more time you can spend on, like, scaling things out, helping your customers, that's going to be a definite win for a lot of businesses.

Yeah. And the truth is this fragmentation is just going to continue to grow. The problem's going to continue to get worse. You see more and more vendors offering integrations, which is a snowball effect because now more and more buyers are expecting integrations, and you're just seeing more and more of that demand. And also with that, Merge is seeing more and more demand for unified APIs.

All right, do you have any final advice then for people who are interested in building these integrations?

Absolutely. I just think before you dive in, you should definitely check out unified API providers out there to see just what it takes off your plate. I do also recommend try building one for fun on your own first. I think you tend to see what happens when the documentation is bad. The APIs don't function as expected, and when you finally onboard your first customer, it's a huge pain. And it was so strong that it's what led me and my co founder to leave our companies two months into Covid to go start Merge. It was a very scary time, but I think the pain was so acute that it was just obvious we had to do it.

Okay. Pain driven development is way more common than I think a lot of people realize. So, yeah, I like the idea that it was so bad that you went off and started your own company to solve the problem. All right, nice. Okay, super. Thank you for your time.

Gil, thank you so much. Appreciate it. Richie.