Hex: Data Project Sharing with Caitlin Colgrove and Barry McCardel

Data science is a collaborative field. Collaboration requires sharing the artifacts that data scientists are working on, such as Jupyter Notebooks and SQL tables. Hex is a platform for improving sharing across data science workflows. Caitlin Colgrove and Barry McCardel are founders of Hex, and they join the show to discuss what they've built. Barry and Caitlin, welcome to the show.

Thanks for having us.

Yeah, great to be here.

You both worked at Palantir, and Palantir was started sort of before tooling for data tools really got good, what did you see at Palantir in terms of the evolution of data tools?

Yeah, so that's actually a really interesting question because I think there's been a lot of movement here just during my career. I've been working in data and data integration for many years now, and I think when I started, what we, and what we saw at Palantir was that most companies had actually surprisingly limited capabilities here. You would go into big companies and their data would be all over the place. Smaller companies were trying to do the right thing but didn't have the bandwidth to really build out a reliable set of data infrastructure. And then over the last 5 years, you've really seen, I think, a significant shift in, in where the industry is. So, you know, you have things like Segment and Fivetran to help manage your data flow. You have Airflow and Snowflake and Transform, excuse me, and dbt to do a bunch of your pipelining. And then all of this is landing in Snowflake, which is a really easy and scalable data warehousing solution. And so now actually what you're finding is Palantir would really excel at was helping people get all their data in order so that they could even begin to do something useful with it. And now we're really coming into companies and finding that they have a lot of this stuff already in place. But they're still kind of stuck on that last mile question of, okay, well, we have all this data and we really want to be a data-driven organization. How do we actually take that data and turn it into something that's useful and accessible in particular to the rest of the people at the company? And I think that's honestly where Hex can have some of the biggest impact.

How did those observations lead to the creation of a product?

Yeah, I think as we, as Caitlin mentioned, sort of we've seen the landscape in the data world evolve evolved a lot over the last few years. And one of the big observations we've made, you know, Caitlin talked about this, this world where organizations are more frequently now having their data in sort of a usable format. One of the big observations and the thing that kind of led to Hex was you've seen this growth in the data space. You've seen a growing number of data teams and these organizations that want to be really data-driven. Lots of new data analysts and data scientists coming in. These are some of the most in-demand roles. The data is now there for them to do interesting things, and they're really interested in how they can take that data and create impact and ROI within the business and not just be like an R&D function where they're just doing interesting, you know, sort of research. And after spending a lot of time in the space and seeing a lot of different data teams and how they operate, one of the biggest gaps that we identified is sort of around this sharing and collaboration bit. You see a lot of data workers that have a hard time both working with each other, so like creator to creator collaboration, but I think more importantly, how they're able to connect that work to the rest of the organization and connecting what they're doing to the questions that they're trying to help answer for the business they're in or presenting it to customers. And so I think that's really where we're focused and what we are most interested in as we're building Hex is sort of how to make those sharing and collaboration workflows in the data space really awesome and something that is supported really well in a first-class way.

What are the frictions around data sharing that aren't handled simply by Jupyter Notebooks?

Yeah, so Jupyter Notebooks are awesome. I've been using them for a long time. I think that format, the notebook format, is great for a lot of things. And we see a lot of data teams that have sort of adopted these as their way of iterating and working on the logic and the analysis they're doing. I think the classic thing we see though is the last mile of that winds up getting really painful. So the story is like the sort of classic workflow we see is, you know, I'll pull my data, usually pulling it locally, like I'm working on my notebook, running it locally. I'll iterate on my logic. I'll get it into a good place, whether it's an analysis I'm building or a model or sort of a deep dive on a question I've been asked. And then if I want to actually take that and share it with others, I wind up sort of falling back to these very legacy ways of doing that. So I might be screenshotting charts. I might be downloading CSVs of the data and sharing them that way, taking these things, pasting them into Google Docs or Slides. Or very often we see people sort of creating these Rube Goldberg machines where they're taking their notebook and trying to get it run on a cron job so it can dump results into a database so we can look at it in our BI tool. And it's like, these are very common sort of friction points around taking the work people are doing and actually making it accessible and useful to the rest of the organization. And we just think there's a lot of overhead there. And there's a lot of interesting room to explore, I think, in how you can help data teams better connect their work to the goals of the organization and make them feel more impactful.

Did the problems that you experienced at Palantir Are those reflective of problems that are affected by the broader industry?

I think so. I think the problems, as Caitlin mentioned a little earlier, I think a lot of the things we saw at Palantir were around data integration and how you can take data from a lot of different sources. And this is what Palantir always excelled at, whether it was in the government space or the commercial work we did. And I think that is sort of a very evergreen problem. And it's been very cool to see a sort of ecosystem of new tools popping up to help solve these things. And I think data integration is a surprisingly broad and deep problem. And I think, I don't think it'll ever truly be solved, but it's cool to see a lot of new tools popping up that I think make it much more realistic for teams to get their data in great usable shape. And I think, you know, it's very interesting to see for early stage companies, ourselves included, you know, even just 5 years ago, if we wanted to set up reasonable data stack for us to be able to integrate and utilize a lot of the data we're collecting ourselves, be it data from our marketing site or product analytics or whatever. That could have been like a multi-week journey of trying to get all of that set up at the least. And today you have a lot of tools that just make it much more, much more seamless and frictionless to get your data in a reasonable shape in something like a Snowflake or a Redshift that makes it very easy to query and access. And now I think the questions are, as more and more companies have that sort of data layer built out and they're layering on things like data quality tools or data catalogs to sort of ensure that the data is discoverable and in good shape. You know, what are you doing with it? How are you making that useful and leveraging it in the business? And I think those are some of the questions that we certainly saw at Palantir and I think a lot of companies are now getting to.

So talking more intimately about Hex, can you describe the workflow for using the Hex product?

Yeah. So our goal here is to have a workflow that's the fastest way to go from your analysis to something that's interactive and shareable. And so really the way you think you start it, the way you would start any analysis, you start with your data. So we have a bunch of really nice features around how to connect to your data, whether it's built-in support for a lot of popular databases. Secure ways to store your credentials. This is one big pain point that I think Hex can add a lot of, provide a lot of added value to. We have a first-class SQL editor. So all of these things really to just help you get your data into your analysis as fast as possible. And then from there, we have a Python-based notebook UI. And this is something that a lot of people use. It's really popular. It was made popular by things like Mathematica and Jupyter. We've already talked a little bit about Jupyter. But for those not familiar who are listening to this, it's basically, it breaks down your code into cells that you can execute independently and iterate on. And the outputs are just right there in line. Super popular because it's a really fast, really nice way for people to iterate on their data analysis. There is some debate here about whether or not notebooks are necessarily the right tool, but we personally love them. We've had great experiences with them and More importantly, we really want to meet people where they're at and really give them the tools that they're already comfortable with. This is one reason why we're 100% compatible with a lot of these open source tools right out of the box so that people can just bring in their existing work and get started right away. From there, Barry talked a lot about the collaboration side of this. Some of that is your typical code-based collaboration, whether it's people working on the notebook at the same time, leaving reviews, publishing, things like that. And some of it is more around the kind of the sharing and permissioning side. So who can have access to this? And really it's going to look pretty similar if you're familiar with something like Google Docs or Figma. That's kind of the style of collaboration that we have in these notebooks. From there, it's super easy to go and create your data application or other sort of interactive artifact that you want to create from your data. Basically what we do is we allow you to take your existing notebook, so you don't have to really rewrite anything, and use the cells and the outputs and the interactive widgets and things that are already in your notebook and compose those into UIs. And from there, you can then go and share that with the rest of your organization. You can control who sees it by these sort of granular role-based permissions. And we also keep a history of all the runs of the notebooks. So if you ever want to see what a particular analysis looked like, back in time, you can really easily go back and see that.

Tell me more about what happens, like what the difference is, the delta between a team that will be using Hex and a team that is not using Hex.

Yeah. So what we see a lot of teams are doing today is they're doing their work in things like notebooks. They're typically working locally and, you know, they're doing really interesting work typically and they're either answering questions for the business or they're developing models or they're working in an analysis trying to understand a key question that's been posed to them. And again, the typical workflow today that we see a lot of people doing is they're taking these, they're trying to collaborate with them through GitHub, which has not traditionally been a great way to manage and collaborate on these notebook analysis-based workflows. And so there's sort of a core pain point there just in terms of making that creator-to-creator collaboration more seamless. And so Hex, we have, you know, real-time collaboration, so you can have multiple people in there looking at the same thing at the same time. We have cell locking, so there's, you know, it's also kind of nice that you're not stepping on each other's work. And then comments, so I can leave a comment and, you know, you can sort of have these sort of review and feedback workflows directly in the artifact that you're working on. And then I think the biggest delta though, and the biggest difference between sort of the traditional workflows we see teams doing and what you're able to accomplish in Hacks is taking that work and turning it into something that's useful and usable for the rest of the organization. And so the big difference here would be instead of the output of our work being a bunch of screenshots that we're putting into a deck or extracting things into a spreadsheet that we're sending around, or you see a lot of like sort of statically rendered PDFs that wind up getting pasted into Slack channels. I can take my logic and instead turn it into a live interactive web app. And that could take the form of something like a report. So, you know, a combination of visualizations and data and commentary around it could take the form of a dashboard that is live and updatable. And non-technical people within my organization could come in and add filters or parameters and and sort of cut analysis in a different way, or something that's much more sophisticated. And I think there's a lot of power that we expose in terms of being able to take something and even build like a data transformation tool or something that's a little more workflow-oriented where I can come in and even be pushing data to other places. And so it's really taking— and I think the thing we're trying to do is help data teams take their work and just turn it into something that's much more usable, and much more accessible and impactful for the rest of the organization. And I'd say that's the biggest difference between that we see and the biggest impact we're trying to create for the teams we work with.

So this ecosystem has a lot of tools that are fairly disconnected. We've already talked about some of them, Snowflake, dbt, Fivetran, Jupyter Notebooks, all these different things. Is it hard to build a product over this shifting landscape?

Yes and no. I think you want to be mindful of how workflows are evolving and the new things happening. And there are certainly some places where I think as things change, you know, we may, we may want to make sure we're keeping those things in mind. But in general, I think especially where we are in the stack, you know, we're focused on really awesome integrations with the data layer. And so for us, that looked like taken the time to develop a really robust data connections backend and a series of plugins that allow us to connect to these common data sources. And so we're a little more agnostic to whether you're using dbt or Dagster or whatever you're using for transformation or whether you're using Fivetran or Stitch for extraction. I think for us, as long as the data is making its way into a common data source like Postgres or Snowflake or Redshift or any of these that we support, you can pick that data up and immediately be productive in Hex. I think there's a lot of opportunity as we look down the road for how, you know, we want to start exposing more functionality through our API, through connecting to other tools that are exposing really robust APIs, and enable workflows that transcend just pulling data out of, you know, a data warehouse and doing transfer, you know, doing analysis on top of it. I think there's a lot of interesting opportunities down that route. And that's one of those areas where I think we're trying to spend a lot of time listening to our customers and what they, what they would want to do with that.

How does a focus on collaboration create difficulties in developing the product?

Yeah, so this is actually something that we have some experience with even, even before Hex. So at Palantir, we actually worked on some real-time collaboration infrastructure. And so coming in, we knew that this is something that we wanted to incorporate in, in the product. I think one trend that we've seen largely in the, in the industry with a lot of these SaaS, new SaaS productivity tools, is that the assumption is that they're collaborative by default. If you have two people coming in and editing the same artifact, it should just work. And people are surprised when it doesn't. So this was, this was something that we were thinking about from the in the beginning, but really real-time collaboration and a lot of the tooling there is still kind of in its infancy in a lot of ways. And so it's a fairly challenging engineering problem. We came in sort of with an open mind, even though we've done it before and did a survey of, you know, where the industry is at in terms of standard algorithms for collaborative editing. And there was really two big ones. So the older, more established one is called operational transforms. And if you're familiar at all with Google Docs, this is what powers a lot of, a lot of that. But there's a new sort of up-and-coming algorithm in this space called conflict-free replicated data types, CRDTs. And there's, and newer products like Figma are based around some of these. So we kind of did an examination of the two of these and what are the trade-offs. OT is very well established. And it handles complexity fairly well. But the downside is there's actually a lot of boilerplate in, in implementing it. And it does rely on a centralized server in order to work. Whereas you look at something like CRDT and there are a lot of advantages. You don't need a centralized server. You can decompose everything into very intelligible single operations on, on data structures. But a lot of the semantics can get kind of complicated around that and A lot of the implementations out in the wild today are still somewhat new and untested and often lack a lot of the features that you can build with OT. So after doing a bunch of research, we ended up taking an approach that was inspired by what Figma is doing, which uses some of the same semantics of CRDTs around how you update your data structures. But we also, you know, we're a SaaS product. A centralized server is part of the deal. So we figured we could actually rely on that to generate and guarantee some, some consistency without relying on, on sort of the distributed nature of CRDTs. Now there's also another wrinkle here, which is collaboration really gets into every single part of your stack. The state everywhere has to be designed with collaboration in mind. And we, we'd already had a tech stack that we were very happy with around Apollo and GraphQL as the main state management tools. And we didn't really want to come in and rip everything out to replace it with a, with a new collaborative library. And we also didn't want to do a whole big single migration over to this collaboration stuff. That can be very risky from a product and engineering perspective. So we kind of wanted to be able to do this piece by piece. And this ended up being a really fascinating project, and there's actually a lot of, of stuff that we learned about how to work with GraphQL and things like that. But we ended up building a library on top of GraphQL and Apollo, which is our sort of main state management in the frontend that allows us to seamlessly swap out individual operations under the hood for their collaborative counterparts. And doing that, we were able over the course of about 2 months to shift essentially our entire frontend from being single player to, to being collaborative with completely seamlessly to the user.

How'd you test those collaborative features? That, that seems like a difficult feature to test.

Yeah, it is, it is really hard. So this is actually one of the parts of our codebase that is the most well-tested. And we actually went and tested it pretty extensively before even touching any of the features. So we've written it kind of as an isolated library. And that library has really extensive unit testing associated with it. So one of the nice things about doing it this way is we can actually test all sorts of different combinations of edits in different orders and things like that. So we can actually be pretty robust in terms of things coming in, you know, what order are the operations going to come in and what's that going to do to the client state. And the other thing is we actually have a set of pretty extensive integration tests to make sure that, you know, the full stack all the way up from sort of the client libraries down to the database are working properly.

What about scalability? Has there been any challenges in building scalable product? Because some of the datasets that people are going to be working with are going to be gigantic.

Yeah, scaling is one of the things that we, we focus on a lot here at Hex. At a high level, the, the power of our platform is also one of our greatest infrastructure challenges. And we're— then this is that we're allowing people to write code, and code gives you the ability to do a huge number of things. So it leads to a lot of challenges around scalability. For example, a user could try to read in 100 gigabytes of data into memory and you have to be able to handle that gracefully. Right now we're leaning a lot on Kubernetes scaling ability in order to handle a lot of this stuff. But there's a huge amount of stuff down the line that we're thinking about building into our infrastructure. Of course, you know, more intelligent sharing of resources, making some of this resource allocation even user configurable. So users can know kind of, they may know ahead of time how intensive their application is, or even sort of tracking and providing feedback to users about, how many resources they're using and allowing that to— using that to really figure out better how to scale the various applications.

Are there any other demands that users have put on the Hex application that have surprised you or made you have to re-architect certain facets of it?

Yeah, I think, uh, Caitlin alluded to this a little bit earlier, but one of the fun parts about building a product that is as flexible as Hex, where we're effectively giving you the ability to come in and sort of write arbitrary SQL in Python. People wind up using it for a lot of interesting stuff. And so I'd say that probably the most fun part of any given day for me is looking at what users have built and having them show us that. And so there's been a lot of workflows that users have sort of wanted to pull Hex toward that we may not have necessarily sort of predicted, but we've been able to support. I mean, like one example, and this sounds kind of simple, but is like, you know, users not just wanting to build applications that pull data and visualize data, but actually then wanting to have that be editable in some way and even enabling their users, so like the people who are using the applications they build, to be able to do some of these write-back workflows. This was not something we had really thought to focus on early on, but as we saw what people were trying to do, it was like, great, we should be able to support this in a, in a first-class way. And so this is where some of the features we're actually working on now around being able to have editable tables and buttons that can do some logic control and things like that. These are the types of things where I think it's fun to sort of let our users guide us a little bit. We still have obviously a vision and a roadmap and a sense of where we want to go. But in our experience, the best way to build incredible products is really listening to your users. Users and having them really show you the value that they can create if you empower them. And so that's been a really fun process so far.

You write about these different categories of data users. So you have the creators, the consumers, and you kind of have the knowledge organization that sits around creators and the consumers. And I just love to hear you talk a little bit more about the roles of the data users and how you see those different people interacting around Hex.

Yeah, that's a great question. And so I think if you could, we kind of back up for a moment and we look at maybe a little bit more of like the, the secular trends here. You're seeing the rise of companies really focused on becoming data-driven and And you're seeing them hiring more and more data analysts and data scientists. And there's just more of these people coming into the industry. This is, you know, some of the most popular things to be studying in undergrad now are around data and data analysis and data science. And so I think as you see more and more of these folks coming on board and coming into companies, the question is, how are these data teams creating impact? How are they creating, you know, a common question we actually hear from some of our customers, especially folks in roles where they're leading the data teams or CTOs, is what's the ROI of our data team? Like, we're hiring these data scientists, we're hiring these data analysts, we're investing in new technologies, like, you know, all up and down the data stack. How are these things actually creating sort of like bottom-line impact for the company? And I think you have this class of, you know, data teams and data workers that now are sort of doing a lot of interesting work, work. And the question is, how are they able to connect that and make that impactful? And so we sort of see these as our creator-type users. And I think the philosophy with them is really enabling them to, to have very flexible tools and very flexible workflows, pulling data, work with it the way they want, and then connect it to the rest of the organization and really bridge the gap between the work they're doing and what other people want to do. And so that comes into another type of persona, which are folks on other teams, you know, you sort of think of the types of folks that might be on a marketing team or finance or leadership where they're very data-oriented, very data-driven people. But they're also not necessarily firing up a Jupyter Notebook locally and pulling in a bunch of Python libraries and doing their work that way. And I think the question is how data teams can take the work they're doing and communicate and share it with these folks in a way that's actually usable and useful. And then there's sort of this overall problem that you alluded to, and we've talked about a little bit around discoverability, which is another sort of pain point you see is, well, great, our data team, you know, they're building these analyses or developing these dashboards and tools, but then this stuff sort of gets scattered all over the place and it's kind of tough to find. And there's this sort of like classic thing we've seen at a lot of customers, like you have a Google Drive full full of slide decks and docs with pasted-in charts, or your Slack channels are just full of links to random one-off screenshots of charts or whatever. And so really trying to build a product that also lends itself well to helping with organization and discoverability, not just for the creators themselves, but a tool where someone who remembers that the data team created a really interesting analysis a few months ago and they want to be able to quickly find it and go back and look at it or refresh it. These things should be much easier than they are today, and that's something that we're thinking a lot about.

How have you seen data workflows change over the last decade?

Yeah, I think when I started my career in data, the teams I was working on and working with were primarily just using Excel. I started in consulting. I was basically an Excel jockey all the keyboard shortcuts burned in my mind. But then what you've seen is over the last, I would say, 10 or so years, especially in the last 5 years, a lot of these people that might have been spending most of their time in Excel are either wanting to learn or picking up new skills around adopting languages like SQL and Python to be able to do their work instead. And these languages have a lot of advantages over some of these legacy workflows in terms of the power of the logic you can build and the flexibility and scalability of the things you can do. And I think this is where you've seen the popularity of things like Notebooks. This is where you've seen the popularity of things like a lot of SQL editors. And I think having spent a lot of time with data teams and working with data teams, this is where we've really seen this gap around making work shareable. And I think there's a paradox of if I move my work from a spreadsheet to something like Python, I get a lot of advantages, but then it just becomes fundamentally less shareable and sort of less useful. I can, you know, share a spreadsheet really easily, but I can't easily share like a Python script or a Python notebook I built with the rest of the business. And I think this is sort of this pain point that we've really been focused on. And there's sort of a question of, as these workflows evolve and mature, and as you have users that are relying on some of these newer technologies to do their work. How do you empower them but also still have that work be really shareable and useful for the rest of the business? And I think there's a lot of room left to explore there.

Are there any other organization-wide frictions that you see in the data analysis and data sharing world that you'd like to address over time?

That's a really good question. I think the problem we're really wanting to solve is around the sharing and collaboration bit. We think that's so core We think the ability to take work and make it into something that anyone else can consume and benefit from is really, really important. I'd say there's a couple other areas that are interesting to us where I'm not sure I would say we're necessarily trying to solve, but we're in some ways, I'm almost looking for people to come along with really good answers here and things we'd love to integrate with. I think one is around data discoverability itself. You know, we're interested at Hex in sort of discoverability of analyses and data projects and work people have done. But I think a little further down the stack, I think there's some interesting energy right now around people that are working on things like data catalogs and metadata cataloging. There's some interesting open source projects that have been spun out of some bigger companies. You have like Adminsyn, that was the tool from Lyft, and there's a few others. And I think we're really keeping our eye on this level, this part of the stack, and seeing what comes out there. Because we think just having better organization of data, better discoverability of data, and better insight into the quality of that data is going to be really helpful for our users as well. And again, I don't think that's something we're necessarily going to bake into Hex. I don't know that that's where we'll create the value, but we're definitely thinking about how we're going to integrate with tools at that level of the stack, because I think that will make the ability to sort of within Hex discover data and analyze data and make sure you're working with data that's reliable and trustworthy all the easier. And that's really, really important.

Caitlin, are there any other engineering challenges that come to mind when you think about the hardest things you've had to build to support Hex?

There are, I mean, honestly, there are a lot of really challenging things. I think even things as simple, like we're a pretty heavy-duty single-page web application that has a lot of performance applications for the front end, stuff like that, that a lot of people I think take for granted or actually end up being really interesting engineering problems with fairly novel solutions for something like Hex. But I think honestly, the thing, the single thing that comes to mind for me most in terms of what we're thinking about for engineering challenges right now, in addition to the ones that we've already talked about around collaboration and scalability, is security. So when you're touching other people's data and potentially hooking up to people's cloud environments and things like that. A lot of these companies, you know, they're data-driven companies and their data is part of what makes them valuable as a company. Sometimes it has additional restrictions on it. You know, it's HIPAA data or financial data or things like that. And so security as a first-class operating principle is, is really one of, one of Hex's core, core engineering challenges, I would say. And this ends up being similar, honestly, at its core to the scalability problem, because what we're allowing people to do is write arbitrary code. And normally, when you're trying to engineer for security, you're trying to prevent arbitrary code execution. You sanitize your input strings and things like that. But that's what the product is. And so trying to architect for security from ground, you know, from day zero has been really important to us. And I think that's going to be one of the largest challenges we face going forward on an engineering side as well.

Is there a challenge of fitting into the existing tooling that the average company already has? So like average company already has their workflows around data science, Jupyter Notebooks, I don't know, Spark jobs and whatnot. How do you squeeze into Or how do you make the product adaptable enough to fit into all kinds of different workflows?

That's a really great question. I think what we've seen is that most companies we're working with have really well-established workflows and infrastructure. I would sort of frame it as like lower down the stack, and we touched on this a little bit earlier in terms of the evolving landscape with things like, you know, the ETL space, what you're using for data warehousing and things things like that. I think what we're seeing at more of the layer we're operating in is a little more fragmentation, to be honest. I think most companies have a BI platform that they're using, be it something like a Tableau or a Looker or a Metabase or a lot of the sort of interesting sort of core BI tools. But in terms of data science tooling, it's still a little up in the air. You do see a lot of people using local notebooks. You see a lot of people with very different patterns in terms of how they're they're managing these. And so I think that's one of the places where we see it as an opportunity to both build a solution that works really well for a lot of people, but then as you mentioned, also integrate well with a lot of the things they're doing. And so as an example, full compatibility for us with the existing file format standard around Python notebooks is really important because we want people to be able to bring their work in. We want people to be able to get their work out and have it be very compatible with different tools they might be using. You also mentioned Spark. I think that's a really good example of, you know, there's companies that haven't adopted that technology at all. There's other companies that have basically built their entire business around it. That was one of the things that we built out over the last few months is first-class support for Spark. And, you know, that's basically an optional thing that companies can turn on and utilize if that's important for workflows, or you can completely ignore it if it's not. And there's a lot more of that we're going to build over time. And then I'd say the other interesting thing here is just from early on, as we sort of navigated the idea maze here and settled on the approach we wanted to take, really adopting and embracing and investing in the Python ecosystem, which has really exploded over the last few years, especially around data science and data workflows generally. Provides a lot of benefit here. You can, within Hex, just as you could in any other Python environment, effectively import whatever library or package is important to your workflow. And I think that gives us a lot of built-in compatibility and integration advantage without needing to build things ourselves necessarily. If you want to leverage, if you want to pull data from Stripe or something, there's APIs and libraries that already make that very easy to do within the ecosystem. And so I think it gives users a lot of flexibility and it doesn't, I think we then are in a position where we're not as opinionated on how we want users to do their work. And I think that's one of the failures you see with some other, or shortcomings you see with some other types of tools where they feel like they have to build, they wind up having to build everything and every integration and everything they want to do and every feature. I think sort of allowing users to always fall back to that release valve of code and be able to leverage the really rich ecosystem around this stuff is really helpful and it's great for our users to be able to do what they want to do.

Does Hex try to address the problem of data catalog?

That's not something we're focused on ourselves. I think, you know, we're interested in how we can help better organize and have more discoverability around the analytical artifacts themselves, whether it's just an analysis or it's something that's been turned into an app or dashboard or tool or what have you. I think the data catalog layer is an interesting one. And there's a lot of interesting energy. And I think we're going to see a Cambrian explosion of tools at that level. I would say that's one of the types of things in the data ecosystem that we're really eager integrators with. And I think as companies adopt some solutions around data cataloging and metadata cataloging, and even some of the things you're seeing around metrics definition and management, I think that's something where we're thinking about building really awesome integrations with because we think that that will be very useful for our users as they're looking to leverage accurate, reliable data in the analyses that they're doing and the things they're building within Hacks.

How does the Jupyter Notebook experience in 5 years or 10 years compare to that which we experience today?

I think there's, you mentioned Jupyter Notebook and that kind of refers to a very specific thing, a very specific open source project that We love— I've used Jupyter Notebooks for many years and I think it's a really vibrant open-source community and there's a lot of interesting things happening there. I think you're just going to continue to see that core open-source project evolve and there'd be a good amount of innovation around that ecosystem. We're very excited and eager participants in that ecosystem. I think there's then the separate question of where does the notebook format go more broadly? There's certainly a lot of other projects that you're seeing that are adopting that same notebook framework and paradigm and taking it in some interesting new directions that are separate and apart from the core Jupyter project. That concept of notebooks I think was originally sort of popularized with even Mathematica, which I think was well before the sort of IPython project which became Jupyter. And so I just think this is a format that makes a lot of sense for a lot of people. And I think we're going to see a lot of innovation around, I'd say, a couple key areas probably. One is just what we're really interested in, which is making these artifacts much more collaborative and useful. I also think there's a note here of how these things can get more accessible. I think the experience for most people today that are not— I would call them analytically technical, maybe they're your Excel power users up into your sort of entry-level data scientists, it's actually pretty painful to use some of these things today. Like Jupyter, despite it being a really awesome tool, it's still pretty intimidating for a lot of people to sort of get started. You're needing to install a bunch of stuff locally and run a bunch of scripts and environment management is a little bit of a mess. And I think it's the type of stuff that makes a lot of sense if you're spending all day in it or you come from more of a software engineering background, but is not necessarily as accessible for most people. And so this is one of the things that really interests us as well is how you can just take these workflows, strip away some of the overhead and complexity and make them much more accessible for a broader number of users. And again, I think there's this large and growing number of people that I would sort of call analytically technical who are looking to graduate out out of just Excel. And I think the notebook format is one that can do a lot of good and be really useful for them. And I expect to see a lot of innovation in this space over the next 5 years. And we're certainly excited to be part of that and contribute some of our own ideas there.

Any other predictions from you or from Caitlin as well about how engineering or data science more broadly will change in the next 5 to 10 years, the whole data stack?

Well, I'll offer one, which is I think today a lot of the differentiation between different roles in the data world are defined a little bit by technologies and tools. And what I mean by this is, you know, if you sort of go through your typical company that has a data team, you might see people who call themselves business analysts that are mostly working in spreadsheets or BI tools. You might be looking at data analysts, and these are people who are working in things like SQL and maybe some Python. And then data scientists who are working in coding languages like Python. And then data engineers who are sort of doing a different subset of workflows. I think some of this is a little, some of these lines will blur over time as some of the tools make more of these things more accessible and some of the skill sets become more widespread. You know, I, I mentioned this a little bit earlier, but when I started my career, if you were working with data, you were probably working in spreadsheets. And I think today a lot of new folks who are graduating or coming into these roles or who are already in these roles and are looking to learn new skills are picking up some of these new technologies. And it's not so clear anymore where the differentiation between sort of a data analyst and a data scientist and these things live. And so I think we'll see more of a blending of these roles and titles. I think we'll see less of the differentiation being driven by sort of the technologies and languages people are familiar with. And I think we're going to see an explosion of people who can be really, really productive and work with data in more and more sophisticated ways over time.

I think on my end, another thing that, another trend that I see happen, starting to happen right now, and I see continuing over the next, you know, 5, 10 years, and this is coming at it kind of from a software engineering background, is over the last 5 years, I would say you've seen this explosion in people working in Python instead of Excel and a lot of this stuff, but you come in and there's not necessarily a huge number of really well-established best practices, the same way that you would have in certain fields of engineering. Like, I remember when I first learned how to program in Java, reading a book in 2004 called Effective Java, and a lot of that stuff is still very true today in how you think about designing Java programs. What I see emerging from a lot of the data science community today is techniques and strategies similar to the types of things that you would have learned for old school software engineering, but how to do data science well, how to do it as a discipline, how to make your results reliable, how to make them performant, and how to build. Sometimes it's actually just incorporating software engineering best practices into actually your data science workflow. So I think that's another trend that we'll see over the next 5+ years is right now there's this explosion of different tooling. And different libraries and all of these things. And I do think that we'll start to get to a place where there are a lot of well-understood strategies around how to do the same that we, I think we've, which we've seen with data pipelines over the last 5 years. We've learned a lot about how to build stable and reliable data pipelines. I think we'll see a very similar transition with the analytical part of the stack over the next 5 years.

Well, it sounds like a great place to close off. Do you guys have anything else you want to add about Hex or data or anything in general?

No, just that like most of the companies I'm sure that you're talking with, we are hiring and we're looking for amazing engineers who are really excited about this space. So, you know, folks can find our website at hex.tech and we'd love to talk to folks who are really passionate about building the next generation of data tools. Awesome.

Well, thanks for coming on the show. It's been a real pleasure.

Thanks a lot, Jeff.