DC_THURS on Data Science, Notebooks & Collaboration w/ Barry McCardel (Hex)

an odd duck company at Palantir and less reliant on like folks like me going out and solving problems in the field. That was really where I also got a lot of exposure to working with engineering and product teams, including my co-founders and other people I work with now. And saying like, well, hey, let's understand the problems people are trying to solve with data and then build tools for our clients to be able to do those things. And so I would never call myself a proper developer, although I feel like I got a lot more technical at volunteer across a lot of different parts of the stack. But I felt very proud of sort of being able to contribute to a lot of the products we wound up building and deploying to our customers and users.

So, and so in what form did these tools like manifest themselves in? Were these still— I mean, I assume it's a little bit less than a hacky Excel spreadsheet or model or Excel, Excel app, if you will. Did you just Excel App Marketplace, or what did you build?

No, I mean, Palantir, and I'll go into a lot of detail on this, but we had our government platform, which is called Gotham, and then we were, the time I was there was sort of this renaissance in terms of building out a more, what I would call, widely applicable commercial platform. It's sort of, it's called Foundry. I mean, there's now a bunch of stuff on it on the Palantir website, but it's kind of end-to-end, all-in-one data integration and analysis platform. That's actually very cool, and there's a lot of ideas in there that we worked on years ago that now you kind of see in this Cambrian explosion of data tools around data integration and quality and ELT and all that. But I think one, probably the most interesting thing and key thing at Palantir is across the business on almost everything we did, I would say the consistent thing was focusing on unlocking the potential of this really big population of analytical people. Who don't necessarily have CS degrees or even like formal data training. And so you think about a lot of the core user personas for Palantir's products. There's like the intelligence analyst in the government side, or someone on the factory floor building airplanes on the commercial side, or a Marine downrange. These are people for whom they need things from their data. They want to do more with data, but they're not like a data scientist. Like none of these people are just gonna like fire up a Jupyter Notebook locally to go like explore some data. And so I think the consistent thing at Palantir and the thing I loved was building tools to empower them and unlock the potential of their data. And I think when you're in startups, when you're in tech, when you're in this data space, if you live in SF or New York or whatever, it's easy to sort of feel like you're in that bubble of like, you know, if you've worked at one of these big companies, Uber or Facebook or whatever, it's easy to think that the whole world looks like that. Like, but it doesn't. And there's millions and millions of people all over the world who are doing things with data. They're sharp, they're data-driven, they're analytical, and they're held back by not having the right tools. And they're not the people that, maybe have the training or credentials to go work at one of these big tech companies that can build a lot of their own stuff where everyone's really technical. And so that was always the interesting thing to me at Palantir was like, how do you meet those people where they are, but also like treat them like the really smart, motivated people that they are and build a platform that empowers and unlocks them. And I thought that was, when I look back on a lot of the work we did, that's what I'm probably the proudest of. And it's almost like building the platforms that can get them out of that hacky Excel world. You'd be shocked at how many really critical workflows in the world across commercial and government are done through hacky Excel stuff. In a lot of ways, we were trying to build the really productized, amazing alternatives to those things.

And what do you feel is the lowest common denominator for technical proficiency to really like, you know, bring these folks onto a meaningful data platform? Like, do you have a SQL belief? Do you have a Python belief? Was it sort of even lower, lower than, than, than those in terms of sort of denominator? Like, what, what's the mean? What's the minimum skill level you believe these smart analysts who are not truly technical like need to sort of like for us, you know, us as a technical community to embrace them? Like, what does that look like?

I don't have a lot of religion around languages. I think you can certainly talk about different languages as being more accessible. SQL is pretty accessible in its own way. I think what's interesting is just looking at the span of my career, there's a lot of our users at Palantir that didn't know SQL. They're probably capable of learning SQL, but maybe that wasn't even the best workflow for them and what they were doing. There's a lot of visual graph-based workflows that we build at Palantir to help people do investigations and stuff where even writing SQL for that would have been clunky, even if that was an easy thing to do. I think the way we look at it now, and we'll probably talk more about this in the context of Hex, is we're big believers in this population that we call the analytically technical. And these are people who, as I was mentioning earlier, they don't have a CS degree, they don't have a data science background, But they're sharp, they work with data, they're curious, they're trying to solve problems through data. And I think the span of this I usually draw is like anywhere from your like really proficient Excel user up to, you know, a true proper data scientist or machine learning engineer. Like this is a span that is big and growing really fast. And I think that a lot of the things we think about as a company are like, how do we build the tools to let them do their best work? So I think of it a little bit less of like a language-specific thing, and I think these people can show up in a lot of different ways. Mm-hmm, got it.

Great. So from Palantir, you went to TrialSpark. And what was the nature of your role there? What did you learn there about data that sort of shaped your current thinking?

Yeah, well, I had a great few years at Palantir. Felt like I learned a lot and just wanted a different shape of a challenge and to do something different. So I went to TrialSpark. It's a healthcare startup. Focused on clinical trials, which was an area I thought was, was really cool and wanted to learn more about. I was in a pretty operational role, which was interesting. And the mission of the company was really like improving clinical trials through better technology and data. And so I really liked that idea of like taking this kind of hard real-world problem and trying to apply technology and data to solve it. It felt very congruent with a lot of the things I'd been doing. And the interesting thing about clinical trials is that they're all about data. It may not be obvious when you think about it, but the whole point of a clinical trial is producing a data set that can tell you something about the safety or efficacy of a given drug or intervention that you're studying. And obviously this is really in the news now, like literally right now, you know, this week. But even, you know, the last couple years of COVID of like all these clinical trials were all waited on bated breath. You know, what you're waiting for is really the data to come out and the data to be analyzed and what you learn from that data. And so I, you know, that's a really interesting frame. And as I got to know the company and what they were doing, that I thought that was such an interesting thing. And there's a lot of opportunities when you think about the way clinical trials are run to improve them through technology and data. And so I came in this sort of operational role. I worked really closely with our tech and data teams. And it was really interesting because I saw a lot of the same pain points pop up that I feel like I had seen sort of throughout my career to date in terms of collaboration and sharing. We had this amazing, really sharp data team. We had an operation that was really hungry to try to power our work and inform our work through data. And we had a pretty modern data stack. We had— we're using like Stitch, Redshift. We were really early adopters of dbt. We had Looker, but our data scientists were— and analysts were really spending a lot of their time in like random SQL IDEs or local Jupyter Notebooks, and they were doing really cool work, but their ability to share, both to collaborate within the team, but then to share with the rest of the organization was super limited. And so I found that super frustrating, having coming from a few years at Palantir building software. I was sort of like, wow, there has to be a product to help solve this. And so I actually started this journey of Hex as the buyer. I was looking for a product that I felt like could help solve this pain we had around collaboration and sharing around our data work. And helping these really brilliant people we had do more and be more impactful. And so that was pretty formative for me in terms of just both seeing the sort of modern data stack emerge, but seeing those gaps. And yeah, I think it took me a couple of months of shopping, basically. Of like, ah, looking at a lot of different stuff and trying to cobble together a bunch of open source things to kind of me convinced like, man, there really is like a thing to be built here. Hmm.

And what sort of open source things did you cobble together? I'm curious.

Well, you know, I think when you look at the data ecosystem and the, you know, the ecosystem around things like Jupyter, there's a lot of cool open source things people have built. Like I, you know, there's a lot of creativity and a lot of neat things in that space. And we tried a bunch of them. We were hosting our own JupyterHub thing. We had a bunch of extensions on it. And then we tried a bunch of these things around being able to share your work in different ways. There's some really interesting projects that have come out of big companies like Airbnb. There's other stuff just in the community. And these things are all cool and good ideas, and they work in their own way. I think the lived experience of them, though, was a little Uh, greedy, like, you know, like it's one thing for like me or other people on the data team or some of our engineers to be able to like run this stuff and understand it. The overhead of maintaining it was really high. And certainly the overhead of trying to get anyone like not really fluent in this stuff to be able to get any value from it was, was just not worth the squeeze. I mean, I remember going around and installing Python, like getting Python environments set up on one of our salespeople's. Laptop so he could run something. And it was like, I just remember being like, this is not how this should work, right? So I don't want to like rag on any open source, they're all like really cool projects. It was more just like, hey, like we were trying to cobble a lot of it together in a way that to kind of form a product in a way that just didn't really work for us. Got it.

So this is the key insight that drove you to start Hex. I mean, I love it when it's sort of so obvious in a person's career and their sequence of their experience that they've basically lived out their own pain and get to this insight, this key insight that really inspires the birth of the company. And it sounds like you had that moment and that's what led to StartX.

Yeah, I had a lot of those moments. We just talked through the sort of span of my career to date and I think kind of every one of those those times, in every one of those roles, there were moments where I feel like I wanted something like CACs. It wasn't always obvious in the moment. It was not like I was in consulting and being like, "No, I'll start a company around this someday." But looking back, it all was forming that primordial soup that I think kicked us off. So yeah, I mentioned I started this journey as a buyer, a user, practitioner. And it took me a while, I think, to convince myself, you know, this is true with a lot of ideas, I think of like, am I the only one? Am I, am I crazy? Is there some product out here that does this really well that I'm just not seeing? And I think it fundamentally was around collaboration and sharing that consistent pain point and reflecting on all those experiences with data that was a consistent pain point. And I mentioned like, doing this crazy stuff in Excel, like trying to ship that data product to others in version control and iterate and get other people to be able to use this crazy product I built was really painful. Collaboration sharing was really core to a lot of the things we worked on at Palantir. And certainly that was the pain point in my role with the data team at Charles Clark. So I think that that was really the key insight of like, this is a world that this data space is a world that I think needs better tooling around collaboration and sharing. And there's also something there about access. If only a small population of people can work with these tools, you're really going to limit your scope and impact. And that was another thing I feel like I took away, especially from my last experience, was if you only have 2 or 3 people who can really go do meaningful work with data, you're really limiting what you— The impact that data can have in your organization.

Mm-hmm, yeah, great. So just from a structural company building standpoint, how did you decide who you wanted to work with to start the company and how did you find them?

Well, I was very fortunate to work with some truly brilliant people in my career and certainly during my time at Palantir is where I met both of my co-founders. Glen Takahashi was my intern in 2014, and we built a very happy product together at the time. And he and I had been working together kind of ever since. We worked together off and on at Palantir, and then he actually came to Charles Barkley with me. So we worked together there. He was doing a lot of engineering and data engineering there. And he and I actually saw— he was one of the first people I talked to about this stuff. I was like, hey, "This all feels really suboptimal, the way we're doing this." He and I just started getting in a mode of jamming about that and dreaming about it. One thing led to another. Then as we were progressing on that, and it went from, "This sucks," to, "Well, what if we built a thing for this?" As that got more serious, I got back in touch with Caitlin Calgrove, who I had also worked with at Palantir, who brilliant, really amazing engineer and engineering leader. And she was like the first and really only person that we wanted to be co-founder and CTO. And so the three of us started talking about this and it takes a while to like go from, "Oh, that's a cool idea," to like, "I'm going to quit my job and do this." But over the course of a few months, we all got there. And it was almost two years ago now that we sort of sort of congealed and got going on it. But one thing I think everyone who's been working for long enough can probably identify with in some way, like you don't realize in the moment how the people you meet and you work with and you will have an impact on your life, whether it's just a networking point or someone you want to start a company with. Like that's, I think, a piece of advice I'll give to other people career-wise is like, just go places where you can work with really great people. Like, that will pay dividends in ways you will never understand. And I'm certainly the beneficiary of that in working with both of them. They're fabulous people.

Yeah, that's awesome. So, so early days of Hex, you had sort of faced the pain of collaboration, data collaboration yourself through a couple different roles. How did you figure out if there was a big enough market for for this? And like, how did you let us test other customers, potential customers, and get that early feedback to kind of prove to you that this is a big enough, big enough thing to pursue?

I think early on, one of the first things that's always tough to get over is like that, why has no one done this before? Or am I crazy? Am I the only person seeing this? And it took a couple months. And so we spent a lot of time asking other people we knew in the data community. Glenn and I were both in New York City at the time, and so kind of bounced around New York. I remember visiting various friends that are on data teams or that worked with data teams, visiting their offices. And this is before we quit. We were just kind of on the side sniffing around this. And spent a lot of time asking, you know, hey, do you have a similar problem? Seeing, going on site, seeing what they were doing. And it took a while, I think, to figure out the patterns between where it was and wasn't the pain point. What phase of maturity of a company or a data team or the types of problems and workflows are they doing that made this. But it was a couple months of doing that. And I remember visiting one friend in particular in New York and it was uncanny how much their workflow looked the same as ours and the pain points looked the same as ours. It's a really common archetype we see today as well with new customers coming on Hex. They're like using the local notebook or they're using some IDE somewhere. And like their workflow is like, yeah, well, it's like, share this. I like screenshot the chart and paste it into a Google Doc or like export the CSV and I put it in a Google Sheet or I render it as a static HTML file and put it in Slack. Like that is just so common. And I remember seeing that like 2 or 3 times in the course of a month as we were bouncing around and being like, okay, like there clearly is some suboptimality here. And if someone has solved this. We're not seeing it. And I think seeing enough of those was really confidence building that there's something here. We may not know the exact shape of what we're going to build, but there's something here to explore.

And so that was the original, I mean, as I recall, that was very early product features in Hex too. It was like the share. It was like the publisher, the share. We talk about collaboration, which could include other things, maybe many things, and we can talk about that. But I distinctly remember just this export, you know, export like graph and sort of share it in an efficient way, in a digitally like native way. Like that was a thing, and that was like one of the early edges of the Hex product, as I recall.

That's right. And early on, in fact, we, you know, we, we even thought and toyed with the idea of even building like a thinner sort of like just last mile thing. Like we actually kind of assumed at different points that people might edit or create their work somewhere else, like use a Jupyter Notebook and then export an IPYNB and then upload it and then build the app in Hex. And so yeah, that was some of the first features we built was around that sort of like publishing and sharing paradigm. The interesting thing there was like, as we, and then just, you know, you're asking about validating the idea and it's like the natural way startups sort of evolve. I think is like, as we were going out and testing that in the field with users, we did hear though, they were like, "Oh yeah," but they were telling us about all the rest of their pain with both their single-player experience of building and working on these projects and then collaborating. And that's where we started getting pulled, I think, by our users and early customers into that bigger scope of doing this more end-to-end and building that platform. And I think over time, there was just this series of Things were really great. Well, let's experiment with that. Let's build that feature. Let's try that. Let's see if we can validate that. And that sort of bigger thing started to really open up for us. And so we kind of started from that sharing end, but it's really obviously covered a lot more ground since then. Got it.

Well, I want you to tell us about what Hex is and what Hex does since we've sort of like poked around the edges, but But before we do that, I'd like to ask about like your product values and sort of your product philosophy. And 'cause sometimes I think that provides helpful context or additional context and really understanding what makes you tick. And before you tell us sort of like what the product actually does, to better understand sort of why you built it the way you did and sort of what design principles you walked into the problem with, I think would be really interesting.

Totally. Well, a couple, you know, we kind of have 3 core product pillars that we talk about internally. One is low floor and high ceiling. And what that means is a product that is accessible, easy for sort of the, let's call it the lower end of that analytically technical population in terms of their raw technical skills to come in and actually do productive work. But a high ceiling that doesn't arbitrarily limit people. I think when you look at some existing tools, and some of the landscape, you see a lot of like fragmentation where it's like, yeah, like this tool does point click no-code things. And this tool is where I can write some simple logic in something. And then I use this tool when I'm writing like SQL, and then I jump over to this tool if I need to do anything in Python. That's like, the fragmentation is as painful as an individual, it's painful as a team. And so we have that low floor, high ceiling philosophy. It was like, we actually believe that you can't be everything to everyone, but we believe there's a span where we want to build the best platform for the people in that, uh, in that world to both work themselves and collaborate with you. On that note, we talk about collaboration and teamwork. I've talked a lot about that just now. It was sort of a pain and inspiration for Hex, and that's obviously a very core value. Hex is a deeply collaborative platform, both you and I being able to collaborate on a project together, but then being able to share, communicate it. So that's another big product pillar. For us. And then I think there's like a thing too, just over everything we build, that really caring about design and quality as a first-class concern. You know, we want to be proud of the work, and we want to delight our users who are using the product. And so that's another thing we think a lot about.

Well, I have to say the design of the product is something that's been attractive to me since the time I first fell in love. So I think it I think it shows that's our core value.

All credit goes to our design and engineering teams on that one.

Yeah, awesome. Yeah, well, so tell us like what Hex does and who Hex is for so we understand the basic use case.

So Hex is a— we call it a data workspace. It's a platform that allows people in this analytically technical population, whether they're a data analyst or data scientist or analytics engineer, whatever they call themselves, to connect to data, query, explore, analyze, and then share. And, um, you know, we think of it as not just as, you know, a notebook thing or whatever, you know. And there's— we have a notebook paradigm, and obviously there's a lot there, but like, we kind of see this as a much bigger sort of end-to-end platform for being able to do go from exploratory data work to something that's actually useful and impactful into— in an organization. And so when you ask, like, who is it for, you know, we want to appeal to that entire span of the analytically technical. And I think there's an obvious first persona at a lot of customers, which is like a, you know, nominal data scientist who might already be using Python or using notebooks. That's it. That's kind of an obvious, like, first adopter at a lot of customers, but It really fans out quite broadly from that. And I think what we see that's really interesting is there's a lot of people in just data analyst roles, people who maybe never would have fired up a local Jupyter Notebook or have never done that, or they find that really intimidating because they have to go learn Python and to enter, manage an environment. And even further, we see people that are product managers, marketers, salespeople. It's really, really interesting how big this addressable population of users, and it winds up being when you build something they can actually use. And there's a lot of our customers where like they require everyone to learn SQL, basically, like, or like everyone who comes in, that's like a prerequisite, nice, you know, nice to have on the JD. And so if you, if you think about that, and if you extrapolate that out, I do think there's going to be, that's going to be true for more and more companies over time. An organization over time, you're going to have more people coming into the workforce with those skills and interests. And I'm really excited about building a platform that can empower that entire range of people, not just the nominal data scientists, which winds up being a small high priesthood at a lot of places. And I go back to that anecdote I was sharing earlier about there's only 3 or 4 of us who could do these things. We want to build a platform that's much more inclusive. Than that.

Got it. Starting with the publishing analogy, like the printing press of the Bible, you know, get the Bible into the hands of the common people, right? Like, that's right.

The gospel of data, right? Yeah. It's the vernacular of— yeah, that's an interesting analogy. I'd not thought of that one before.

You can use that for free. I mean, it's on me.

Thank you. No one knows that I got that from you, right? We're not live or anything, so that's good.

Well, so, so the Hex platform is really compelling. I've seen the tool. As I mentioned, the design is just awesome. I really respect what you guys have done. I want to hear about some of the new features that you're most excited about because, you know, instead of just thinking of Hex as like a big giant notebook, collaborative notebook in the cloud, that's sort of easy for everyone on your team to like poke at and run and, you know, log into and view and share and and all those collaborative things, you've actually pushed the paradigm in some really interesting implementation ways. So I want to hear about that. And I think you've called some of this Hex 2.0. Tell us about some of the things that you're most excited about in that category.

Yeah. Yeah. So it's been really fun this year, especially I just think as the product sort of, as we built the foundation out, been able to go pursue a lot of these big ideas that we've had for a while. And last month we launched X 2.0, which has had some pretty fundamental reworks of some of the ways that these workflows operate. I think one of the big tentpole features there was our new execution model. And so I think for anyone familiar with notebooks or these workflows, they've probably run into this frustration. And I think it's best encapsulated by Joel Groose's infamous "I don't like notebooks" talk. I hate notebooks. Yeah, "I hate notebooks," whatever it was. And as someone who's used notebooks for a long time, I love notebooks. I think they're great. I think they're really cool. But you look at Joel's talk, and then you reflect on your lived experience of using them, and you're like, yeah, he's right. Problems in here. And I think for us, it was— it's been an interesting journey, sort of both embracing that notebook paradigm, like, hey, this is a really cool way to work with data, along with like also acknowledging that like, well, there's a lot of room for this to evolve and grow. And so in Hex 2.0, we introduced a new DAG-based reactive execution model. And so to unpack that a little bit, it's— we parse out the code in a given project, the SQL and Python in each cell. And we are doing some analysis of the code to understand the dependencies between cells. So each cell sort of has a sense of objects that return. So if it's like a query cell where you're writing SQL, it's returning a DataFrame. Python cells, you know, similarly can return DataFrames as objects. They can also just return parameters or other types of objects. And then we parse the code to understand the dependencies. And so that actually forms a DAG, a directed acyclic graph, which is obviously a concept that's very common. And in our blog post actually about Hex 2.0, one thing I point out is like a lot of people are familiar with a DAG-based reactive execution model because of spreadsheets. So going all the way back to the spreadsheet stuff, spreadsheets operate with a reactive DAG. Like if I update cell A1 and I'm referencing it in cell F5, like —that will just— there's a reference, it knows. You can't do circular references, so it's an acyclic graph. It's kind of funny how many people in the world use a reactive DAG-based execution model today in spreadsheets. And so this isn't necessarily a new idea we had, but it's taking that paradigm and applying it into this world. And so this execution model solves, I think, the 3 big problems around state in notebooks, which is reproducibility, So when I update something, knowing what's going to update, knowing that it's going to update reliably, knowing that my logic is going to rerun in the same way every time. Interpretability, which is I as a notebook user, both for myself and while collaborating, it can be very hard to understand what's going on, like especially the way people wind up writing code in notebooks. It's not necessarily always like really, you know, well organized and then nice functions or whatever, like. And so by having this DAG that you can actually visualize, and so in Hex 2.0 we actually have a graph UI that you can inspect, it makes it really easy to actually interpret and understand what is— well, how does my logic— this cell connects to this cell connects to this cell. I can see the whole dependency tree. It's really, really nice. And then performance. And I think it's an ironic thing that we talk about in Hex, you can publish your work as a data app that anyone can use. Well, the way we did that was, okay, cool. When you publish it and someone's using it as an app, we'll rerun the logic top to bottom every time. And that helps solve that reproducibility problem around like, well, it's just going to run top to bottom. And so it's easy to understand what's going to happen, but it has a performance implication. It's also more like it's going to run all your logic every time. And so by doing it through the DAG, you're only rerunning downstream logic that needs to rerun when you update an upstream node. And so it's a really great model for that. So again, solves the problems of reproducibility, interpretability, and performance. And this is great for advanced users who have been using notebooks for a long time. Like, I've been using notebooks for a long time. This is a better way for me to work with that. And that's really awesome. But it's really also great for this bigger population of people that we've talked about a few times that are like exactly the people for whom, like, they get a weird kernel thing or they get, you know, dependency error and state, weird state stuff that they can't understand. You know, exactly the people for whom like notebooks in that paradigm will just drive them up the wall or discourage them. And it's like a barrier of entry to these workflows generally. Like, if even if you can get a local Python environment and get a Jupyter Notebook running, like, these are the types of things that just make it really, really intimidating. And so this model being more interpretable and approachable, I think, is really, really cool for that bigger population of users and something we're really excited about. Yeah, that's awesome.

And what's the response been like to this feature so far?

Well, great. I mean, you know, we launched it, we had the blog post explaining it, and I've gotten just kind of a flood of both people like really responding to the problem, like, and even Joel Groves, I think, tweeted about it, like, you know, like, like, you know, the patron saint of there's a lot of problems with notebooks. I think, you know, clearly resonated with people who have felt that pain. And I think—

Did he say this is cool? This is cool or this sucks? What did he say?

No, no, he tweeted about it in an admiring, in a good way. I forget how to speak for a good way. Yeah, kind of like, I think for him it's cool to see, I don't wanna put words in his mouth, but I sense it's very cool for him to see people taking the critiques he has outlined and sort of doing something with them and saying, well, let's actually try to address those in a really forward way. And then I'm really proud of our team that the implementation of it is really cool at the technical level, the way it actually works, and then the graph and the UI around it I think is really sharp. So the response has been great. It's well received by our users and the foundation for a lot of other stuff we want to go and do. That's great.

Very cool. I also wanted to ask you about this other feature that I read about, which is the DataFrame SQL. Oh yeah, DataFrame SQL. Because that was new to me. How does that work?

That's a cool one. So in a notebook traditionally, you, you know, it's actually a notebook traditionally, it's really just all Python, right? And you can like write some SQL if you want to query something in a structured way, or it's like, okay, I'm gonna like roll my own like database connector, you know, import SQLAlchemy and do that. And then like write a SQL string and shove it into this Python call. And it's like, it's really clunky. It's not great. So many months ago in Hex, we introduced first-class SQL cells. So it's a nice SQL, basically a SQL IDE within a notebook. It's really awesome. That SQL cell returns a DataFrame that then you can use downstream and other in, in your Python. DataFrame SQL kind of says, well, we're going to treat every DataFrame in the scope of a project as a effectively a table that can be queried. And so the results of a SQL cell are then actually available to be queried by another SQL cell. So you can write SQL on SQL, which is sweet because it's better. You see a lot of the SQL queries people write, and they have these CTEs. So for people who write SQL, it's like WITH AS statements, where you're kind of creating these subtables and then doing it. So DataFrame SQL is sweet because you can kind of unbundle that and actually treat each of those things that you would previously have as a CTE as its own query. Join things up, and you can also have data frames created by Python, uh, also available. So even if I'm doing something in Python or Pandas, you know, I'm creating a data frame that's also available to be queried. And so we kind of treat every data frame in the project as part of like a meta database. And so what that means is that you can actually create a whole project that's end-to-end SQL, which is what a lot of our users do now, or a whole project that's end-to-end Python. Or a mix of the two and whatever you want. And so that kind of flexibility is really cool and it kind of abstracts a lot of it away and combined with the DAG model too. And so almost a very functional node-based thing where it's like, okay, this cell creates a DataFrame that can be queried here and done here. And it's very, I think, intuitive to a lot of people, especially those who are coming to this, not necessarily from a really deep data science, long-term notebook user background. Yeah, got it.

And what about the implementation of a feature like this? How did you do that?

Yeah, we are— we actually looked at a bunch of things over time, and one of the things that really sparked us was we discovered a project called DuckDB, which is an in-memory OLAP data store, and it's pretty cool. It's, uh, you know, it's still pretty early, they're like version 0.3 or something like that, but great team. We're actually supporters of the DuckDB Foundation now, and it was a really cool piece of technology for us to be able to work with and use to enable that. So we were able to take these data frames and use DuckDB as then effectively the query engine for those, and it's very Postgres-like syntax, so it's pretty vanilla as far as SQL syntax goes. Yeah, it's really cool. Very cool.

Since we're talking about sort of the under the covers in the product. What's an example of a more daring architectural decision that you'd say you've made in building Hex?

There's a few. I think the bet on the Hex 2.0 model was kind of a big one. Deciding that we wanted to walk away even from some of the traditional precepts around notebooks and how those work.

You mean with the DAG, the DAG execution?

The DAG, yeah, the DAG execution model. And actually, it was an interesting journey because we actually tried a few different implementations. And I think for us, it was even sort of believing that we could show a better way to do this, even if it was different than how people might be used to working. And so one of the cool things about Hex, I think, is that if you're an existing notebook user, like if you're familiar with tools in the space, Hex will feel familiar and powerful. But then we open up these new ways of thinking and working about things with things. And so Candidly, there's certain things that you could do in Python, like a traditional Jupyter Notebook, that like won't work with this execution model. Like there's ways of like certain syntax and things that, you know, our parsing won't properly do. And like, yeah, okay, well, that's the trade-off. We might, you know, get pissed some people off. But for us, that felt worth it to have this better model. And so, you know, there's trade-offs with all of those things. And then I think there's some interesting architectural things as well, just in terms of the bigger platform, how we're scaling the compute and all of that. That's probably a talk for another time, but yeah, there's a few big bets we've made. I'm really proud of that.

Got it. Cool. And how would you say Hex fits into a cloud-native world? I mean, it seems fairly obvious to me. I guess from knowing a little bit about the tool and the way it works, but talk to us about the bigger vision that you might have there.

Yeah. Well, I'd say Hex is fundamentally built for a cloud-native world. I mean, it's cloud-native in a way in that it's hosted, it lives in the cloud. I think when you think about cloud-native, there's probably a couple lenses there. There's cloud-native in terms of collaboration, which cloud-native just means online and collaborative, right? And I think Hex was built from day one with that assumption of it's going to run in the cloud, it's going to be easy for people to collaborate in the cloud in that way. It's very cloud native from that aspect in the way that something like a Google Docs or a Figma or something is cloud native. And then I also think there's another angle of this, which is cloud native in terms of the way data workflows are going. You think about what does it look like to have a cloud native data stack? Well, it's a data warehouse like Snowflake. Its cloud-native data processing tools like PyCharm or dbt. And Hex works really well with all of those things and also sort of assumes that people are going to be working with data on that scale, at that cloud-native scale. And so I think we're just scratching the surface of what cloud, you know, cloud-native, cloud-first world looks like for data. And hopefully we can make some small contribution to that in our own way of what that means. Got it.

Great. Well, we're almost out of time, but I want to ask you a couple more questions before we let you go back to building all that Hex goodness. What's your bigger vision for the company? What's your thought on where the product heads and where the company goes next?

We have a lot of ideas. A lot of things on the roadmap in our mind. I think one of the hardest parts is just focusing. It's picking out what are the few things we want to go really invest in. And there's always a tension, I think, between building the immediate things people are asking for and then stepping back to those bigger bets. No one really asked for Hex 2.0 or the DAG execution model in the way we built it, but people were asking for a lot of other things that we felt like, well, that's actually the real way to go. Fulfill that or to solve those problems. And so there's a number of other things like that on our minds of things we want to go and do and invest in and big bets to go take. I think it all comes back to this mission we have about empowering everyone to do more with data together. Should be, you know, how do you help increase the access to these workflows? How do you help empower more and more people to do amazing things with data? And how do you help turn that into a more collaborative exercise. And again, I feel like we're just the tip of the iceberg, or whatever metaphor, scratching the surface, whatever tourism you want to use about, about going and doing that. And so, you know, we've got, we've got a lot of stuff we're excited about. And I think even between now and the end of the year, I have a couple of big things I'm excited to get out the door. Great.

What surprised you most about working in this community so far?

Surprised me most? I don't know. I don't know if it was particularly surprising, but I was certainly excited and grateful for how warm and collaborative everyone is, even folks that we kind of maybe compete with. It's a crowded space. Everyone kind of competes with each other at some level. It really feels very authentic, and everyone's sort of just excited about cool stuff. I talk with people all the time that are even adjacent or whatever. And it feels like we're at this moment in data right now where there's such an explosion of new ideas and things. And some of these things will work and become really big. Some of them won't. And that's all kind of baked in the cake. And I think both within Hex and then in the broader community, it feels like there's just a lot of excitement and energy to just be exploring and trying new things. It feels very collaborative. And so I'm very grateful for that. And I'm going to do what I can to try to keep that going.

Yeah, that's great. Well, you haven't even been to Data Council yet because we've only met since the very early days of the pandemic. So yeah, just wait. This is the whole sort of function behind Data Council. Not to be too self-promotional, but I'll get a word in Edgewise anyway. Can't wait to get the community together in Austin.

It's going to be awesome.

There's so many amazing tool companies that have come out of that collaboration and that community. So, couldn't be more excited to support that, exactly that sort of notion of collaboration that you're touching on.

I'm excited about that and I think you do a great job helping facilitate that through the work you do, including Data Council, which I'm excited. I'll be there in Austin, so anyone else who's there, I would love to meet you. I'd love to hang out, would love to talk about this stuff. And I'm really looking forward to that. Yeah, very cool.

Looking forward to having you there. So Barry, we'll let you go, but I want to just before we do, make sure that people know how they can try Hex and how they can reach you if they have questions or comments.

What are the two best hooks there? Yeah, well, hex.tech is our internet website that you can go visit and get access to the product. And then my email, if you want to reach out, is pretty easy to guess. It's barry@hex.tech. Great. I'm occasionally active on Twitter. My handle is @barrold, B-A-R-R-O-L-D. But email me, find me on Twitter, and I'm pretty reachable for folks in the data world that want to talk about this stuff. I'm always down to chat. Would love to connect with anyone virtually or in person now that such a thing is possible again.

Yeah, great. Awesome. Well, thanks for that, and thanks for joining us, Barry. It was really great to have you here. Yeah, likewise.

This was fun. Thanks for hearing me out. I hope I wasn't too boring.

No, it was great. Great chat. Cool. I'll see you, Pete. See ya. And to our audience, thank you for tuning in. I want to remind you that next week, November 11th, We'll be talking with Rick Lammers from Orchest on data science pipelines. So don't forget to join us for our next DC Thursday next week. Thanks again to everyone for listening, to Barry again, and we'll see you soon.