Pachyderm's Kubernetes-based infrastructure for AI

Bandwidth for changelog is provided by Fastly. Learn more at fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode servers. Head to linode.com changelog this episode is brought to you by DigitalOcean. They now have CPU optimized droplets with dedicated hyper threads from Best in class Intel CPUs for all your machine learning and batch processing needs. You can easily spin up their one click machine learning and AI application. This gives you immediate access to Python 3R, Jupyter, Notebook, TensorFlow, Scikit and Pytorch. Use our special link to get a 100 credit for DigitalOcean and try today for free. Head to dot co changelog once again dot co changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're PracticalAI FM and now onto the show.

Well, welcome to Practical AI. Hey Chris, how's it going, man?

Pretty good. How you doing, Daniel?

Doing?

Doing really good. I'm really happy today with, with the conversation that we're going to have because we're going to be talking to my, my old colleague and still great friend Joe Doliner, or as I call him, jd. Welcome, Joe.

Hey, Dan, it's great to be here. Hey Chris, it's great to meet you on your show.

Great to meet you too. Yeah, thank you so much for, for joining us.

Thank you for having me.

Yeah, why don't you give us a little bit of background about what you're, you're currently involved with and, and how you got there.

Yeah, absolutely. So, as you said, I'm Joe Doliner. Everyone calls me jd. I am the CEO and founder of Pachyderm, which is a company that builds data science tools that we'll be talking about today. Before that, I've worked at a number of startups. Probably the most relevant one to this conversation is that I also worked at Airbnb as a data infrastructure engineer, basically just managing their AI and data infrastructure for the company. And so I have a lot of experience on the infrastructure side of data science, less so as an actual practitioner. And so that's most of what we're going to be talking about today.

Awesome. Yeah, that's perfect setup. I think that we've done a lot of talking about AI, but we really haven't got into a ton of infrastructure stuff yet, I don't think, have we. Have we, Chris?

Not really. And it's. I think this is an episode long overdue. And it's just to note to the listeners I know you had that you had previously worked with JD at Pachyderm. I have not familiar with Pachyderm as a, as a newbie, so it'll be an interesting conversation for me having a couple of experts on here and I'm going to ask all the stupid questions.

Okay, well. And you know, he's not my inside man.

That's true.

Dan might be, but Chris definitely isn't.

Yeah, yeah. Full disclosure, I might be a little bit biased, but only I don't officially work for Pachyderm anymore. Although I am a huge fan, I am actually using Pachyderm on, on my current project. So I'm a huge fan and have that bias. But I'm excited to dive into the details and have you learn a little bit more too, Chris?

Yep, absolutely. Over the time that we've known each other since we first met and you've been talking about it, I've adopted it. I have a long way to go to catch up to where you guys are in terms of using it as a tool. But as a beginner, it's definitely something I'm interested in. So I can't wait to hear more from jd.

Definitely.

Yeah.

So with that jd, why don't you give us just kind of a high level overview of what Pachyderm is and kind of the needs that it's fulfilling or what it's trying to do for data scientists and people working in machine learning and AI.

Yeah, absolutely. So Pachyderm is basically designed to be everything that you need to do. High level production, data infrastructure in a box. And so what that means if you're used to, you know, doing AI workloads in, you know, jupyter notebooks on your laptop, or maybe just in Python directly using something like TensorFlow, something like that. Pachyderm is not in any way saying that you should stop doing that. Pachyderm is just giving you a way to take that code and deploy it on the cloud in a distributed fashion so that, you know it's going to run every single night or, you know, hook it up with its processing steps so that you can have everything sort of going in a pipeline end to end. And this is what companies turn to when they sort of need to make that leap from a model that's on somebody's laptop to something that's like a core part of their business that's going to run every single night. This sort of all came out of my experiences at Airbnb, where I was basically trying to make a platform that did that for our data scientists. And while I was working there, I had a couple of sort of novel ideas for what I thought that the world of data infrastructure was missing and what I wanted to bring to it. So the first really unique thing that we did with Pachyderm is we needed a way to store data. So we have a distributed file system. It's called the Pachyderm file system. If you're familiar with the Hadoop ecosystem, this is probably something pretty similar to HDFS or Tachyon or something like that. What's different about our file system is that it's capable of version controlling large data sets in addition to storing them. And so you can have your training data set. It can be terabytes of data, and this data is constantly coming in from your users on a website, from satellite imagery or something like that. And the Pachyderm file system will actually give you discrete commits, like in Git, where you can see, okay, this is what my training data set looked like a week ago, this is what it looked like a month ago, and things like that. And what's really important for AI, that is not only do we keep these different versions, but we actually link them to their outputs using a system that we call provenance. And so at any time when you've trained a model in Pachyderm, you can ask the system, what is the provenance for this model? And it'll trace you back to all of the different pieces of training data that went in it, into it, and all of the different pieces of code that went into training this model so that you can, you can basically see where it came from and you can reproduce your results. Does that make sense to you guys?

It does. I'm going to dive in, since I'm the newbie on this, and, and ask so kind of. And I'm asking this on behalf of the listeners and partly for myself. First of all, quick question. Is it a proprietary system or is it open source?

This is all open source. We do have an enterprise system that goes on, on top of it. And I'll talk to you later about what, what features are limited to the enterprise system. But nothing that I've talked about up until this point is in that this is all open source. You can download it yourself.

Okay. And to kind of wrap our heads around it a little bit. You kind of mentioned the file system and versioning and this, what sounds like a kind of a feature called Providence, where you can go back and do that. Could you kind of describe for someone who has never heard of Pachyderm, kind of what the feature set is and what kind of a typical use case might be so that in their own shop where they're doing data science, they can kind of figure out how it fits in with what they're already doing?

Yeah, yeah, absolutely. So I think it's. I think it's easiest to sort of focus in on a use case here. So one that I can talk about very publicly, because it was a public competition, was the Department of Defense was, until recently, running a competition where they were basically having people write image detection algorithms for satellite imagery that they had. Right. So they had a bunch of satellite images that they had taken, and they wanted people to write models that would detect. This is a hospital right here. This is a school, this is a bus, things like that. Interesting AI problem. Also an interesting architecture problem for them. Right. Because they have people just basically throwing code at them through this web interface. And they need to take that and run it through their pipeline and get results out the other end and give those to the users. So the way that they set that up in Pachyderm is first they spun up an instance of it and they deployed it on AWS. They used as the backing store. They used S3. So ultimately all of this was stored in Object Storage, which made it very, very easy for them to manage. And then they loaded all of the satellite images into the Pachyderm file system. And so that's, you know, you can. You can get stuff in there in a number of ways. You can get it in there directly from Object Storage. You can push it over HTTP. I'm not sure exactly which one they used, but from there they now had a system where all of the data was just sitting there in different versions. They could update it and have a new version. And then anytime that a user's code came in, they just deployed a new pipeline on Pachyderm, and that would then slurp up all of those images and process them in parallel. And out the other end, after some processing would come just a score report that they could report back to the user. And that might include your code failed on these five images. So you don't get a score. Or it might be your code succeeded on these five images and here's how accurate you were. And it would get them full reports about, like, here's what you did well on, here's what you didn't do well on, things like that. Does that answer your question, or do you want to know more about sort of specific features within Pachyderm?

No, that does help a little bit, I guess. As a follow up, you talked about file system and its ability of versioning. Are there any other kind of high level, like key things that you want to name that you really can't use pachyderm without considering those features?

So in terms of the file system that really basically covers it, it does basically all the standard things that you'd expect from a distributed file system, plus the versioning and provenance component. And that's really the only quirk to it. Now, on the processing side, things also start to get interesting, and here is where we need to start introducing maybe a few jargony words that I will explain. So one of the sort of key things that we use in Pachyderm is Containers. And I'm sure most, most listeners at this point have heard of the company Docker, which has been a very successful Silicon Valley company, and they make this thing called a container, which is basically just a standard way to ship around code, right? Think of the problem that you've had where you write some script in Python that trains a model, then you send it over to your friend and they've got the wrong version of Python, or they've got the wrong version of TensorFlow installed or something like that, and it's all incompatible. A Docker container is a way to ship code that's going to work anywhere, regardless of what the user has got installed on their machine, or regardless of within the cluster. Pachyderm's processing is all built on Docker containers. And so what that means is that you as a, as a data scientist, when you want to productionize your code and take it off of your laptop and into the cluster, then all you need to do is package it up into a Docker container, which means that there's a little bit of a learning curve there to understand the tooling of Docker. But once you've got that, you as a data scientist, are now completely in control of the environment that your code runs in and all of the dependencies and everything like that. And so once people grok this, it's actually very, very liberating. And the reason that I wanted to build this on top of Containers was because when I was at Airbnb, we would have these problems all the time. Where a data scientist would come to me and they'd written some new piece of processing that they wanted to be in the company's pipeline, could be a machine learning model, or could just be something as simple as data cleaning or something like that. And they would send me the Python script and then I would realize, oh, this isn't quite compatible with what I've got on the cluster. And we didn't have Docker containers there, we just had one big monolithic cluster. And so if we didn't have the right versions of Python installed, I actually would have to either redeploy the entire cluster just to run that one user's code, which was very untenable, or I would have to have them change their code to use different versions, things like that. And so it was this constant back and forth where the data scientists couldn't quite use the tools they wanted. Our infrastructure people couldn't quite, like, maintain a cluster with a consistent set of tools. And so I had this aha moment when I realized if these guys could just use Docker containers, then this impedance mismatch would totally go away and we could both do our jobs a lot more easily. Does that make sense, Chris?

I was just going to say, like, following up on that. So it's kind of like the. Whether you're using Python or R or, you know, Java or whatever the different tool you're using is, or the language you're using, essentially these containers unify the way that you treat each processing step. Would that be an accurate way to see it or to say it?

Absolutely, yeah. So it allows us to basically handle the infrastructure the same way, no matter what code it's written in. And so we have a lot of companies where one of the things that's really appealing about Pachyderm is that they, all of their data scientists just know different languages and they're looking for some sane way to have everybody writing code in their own language and pull it, tie it all together into a system that they can understand, and Pachyderm allows them to do that. Now, the key thing about this, of course, is that because we have the provenance tracking, like, you can still see the fact that, like, oh, this data came, followed through all of these steps and came out the other end. Even though, like, one step was Python, one step was Ruby, one step was Java, one step was C, and you didn't have to write any special tooling within those languages to track the data.

Yeah, that's awesome. So I'm going to kind of pose a problem. I want to see if you would kind of go about things the same way as I would. JD so let's say that we have a, you know, we have a jupyter notebook and I like how you kind of brought that up before because that's where a lot of data scientists kind of start out. So let's say that Chris and I have been working on this jupyter notebook that has some pre processing for images and then we train a particular model, let's say in TensorFlow, and then we output results and then maybe do some post processing and to test it out, we've just kind of downloaded like a sample data set of images locally. And then we've kind of proven that, yeah, this is like a good way that we think we should do this kind of in this Jupyter notebook. So in order to, for us to kind of get this, that scenario off of our, you know, off of our laptops and into, into Pachyderm, what would be the things that we would need? What would be the steps that we should do, both on the data and the processing side?

That is a great question and I think will, will be a really illustrative answer. I'm going to sort of try to answer this with, rather than jumping straight to the like. So here's the end state of this where I think it's like you're using all of the Pachyderm features. I'm sort of going to build it up piece by piece, which is how we recommend data scientists to do it. So the first kind of problem that you need to solve when you want to put a jupyter notebook into Pachyderm is the fact that jupyter notebooks are meant to be interactive. Right. They're meant to have a user like opening up the browser and actually like clicking the run button and stuff like that. And so the first thing that you can do is you can actually run sort of Jupyter inside of a Pachyderm service and you can just run Jupyter notebooks all by themselves, but they can't just turn into a pipeline that runs without any human intervention. Right. Because Jupyter isn't designed that way.

And so like in an automated and triggered sort of way.

Right? Right. So the first step to do is just to extract the code from jupyter. I'm pretty sure jupyter makes it very easy to export as a Python script at this point. And so you would, you would do that and then you would put that in a Python container with whatever Dependencies you need. And to start, I wouldn't even tease apart these different steps. The pre processing, the model training and the post processing, you could just do all of those in one container. And you wouldn't even necessarily need to parallelize the data because if it was running on your laptop, it could probably run on a beefy EC2 node as well. And so that process I think would take you. You know, there's if, if you had pachyderp set up to begin with, you could probably do that in 20 minutes. And then you would have gone from a system that you can run manually on your laptop and edit to a system that now runs every single time a new image comes into the repository or you change the code or something like that. And also of course now it's deployed on the cloud, so you can easily throw a GPU in there if you want. You can easily throw more memory at it and stuff like that. And so now you have sort of the first step of a productionized pipeline. Now the next step is figuring out which of these steps does it make sense to tease apart so that maybe their outputs can be used by other steps. You know, in the future you might want to do the same pre processing and then train multiple different models and then do the same post processing on them or something like that. And so I would separate out the pre processing step, the training step and the post processing step into their own individual pipelines. And so now I've got a chain of like three steps and each of these is doing something different. And now I get the opportunity to sort of optimize each of these steps individually. Right. So the pre processing step, for the most part, the pre processing steps that I've seen can be done completely in parallel. Right? You're doing things like cleaning up the images, you don't need to see all of the other images to clean up one image.

Parallel as far as like in the sense of distributed processing, like processing things in isolation.

Exactly, exactly. So that's another of the like sort of important, important things that we get from a container is that it's very, very easy for us to scale that up. Right? So you can say, I need to process all of these images. Here's a container that does it. But don't just spin up one copy of this container. Give me a thousand. And so you're now cranking through a thousand images at the same time rather than one. And so you'll get done much, much faster and you can handle much, much bigger loads. So I would do that with that step, the training step. Training. Making. Making training happen in parallel is definitely a much more complicated question than making something like pre processing happen in parallel. So normally we would still keep that as a non parallel thing because your code needs to see all the data to train on it. If that is not true, if you really want to start parallelizing that, that is when you want to start looking at things like kubeflow, which we integrate with as. As you know, Dan, although we're still working on making that integration better. And then the last step, the post processing step, that one could sort of stay as is unless you were anticipating having a lot of things that you wanted to post process in parallel. So for example, when the DoD did their pipelines, theirs is all designed around the fact of we have one data set, but we have, you know, thousands of different people submitting models that they want to get tested. And so actually the post processing step could be pretty expensive because they were just doing it for so many different entries. And so that was happening in parallel as well. From an infrastructure perspective, that's basically the idea of these pipelines is that when you segment these steps off into little pipelines, you then get complete control over the infrastructure on a pipeline by pipeline basis. So you get the ability to say like this one needs to run in parallel with 1,000 copies of the container up, and each of those containers needs to have a GPU accessible to it and this much memory and stuff like that. And this one over here is not doing it really much at all. So like it just needs one container and we'll fit that in somewhere and the system sort of automagically figures out how to make all of this work with the resources that it has.

Okay. Hey JD that was a great explanation. As a beginner, I have a few questions I'd like to follow up with. First of all, you mentioned kubeflow. So I take it that Kubernetes is part of the architecture that you're deploying onto.

Yes, I guess I jumped the gun a little bit on that one, mentioning kubeflow before kubernetes. But yes, this is, I think now when we need to bring in one more jargony word, and this will probably be our last infrastructure jargony word, which is kubernetes. If you've heard of Docker, you've probably heard of Kubernetes as well. Actually at this point, I think if you install Docker, it just has Kubernetes built into it. You should think of Kubernetes as kind of the Puppet master for your containers. Right. So a container is a really, really good way to deploy a single piece of code, like a program. It's literally just a process inside of a box. To deploy complicated distributed applications. You need to deploy a bunch of programs on different machines and make sure that they can all talk to each other and that they have the right resources and everything like that. And that's the piece that Kubernetes handles. So Kubernetes allows you to speak in very high level terms. That were a lot of the terms I was talking about. Pachyderm speaking in of bas. Basically being able to say, I want you to make sure that there is a copy of this container running somewhere. You have a thousand machines, you have the code to run. Just make sure that this is always up somewhere and I can talk to it consistently when I hit like this IP address or something like that. And Kubernetes will figure all of that out in the background for you. And you know, it can be instead of one copy, it can be a thousand copies. And they can have specific infrastructure requirements like GPUs and stuff like that. And Kubernetes just solves all of that and deploys all these containers. And so that's how we accomplish that with pachyderm is we basically just take these Kubernetes semantics and then augment them with knowledge of the data that needs to be processed and capture how that data gets processed and where it goes.

Gotcha. So just to kind of catch up a little bit and make sure I'm on the right track, you have Kubernetes deployed for infrastructure and you're deploying pachyderm on top of that. And you have the file system that it brings with the versioning and your capability for provenance tracking. And you've talked about the pipelines and stuff I take just to ensure that I'm on the right track. I assume that the data is in the containers that you're deploying specifically.

Yeah. So that's where it starts to get interesting. The data is in the containers, but it's kind of ephemerally in the containers, because containers themselves are kind of ephemeral. Part of the point of a system like Kubernetes, and the reason that you give it 1000 nodes to operate on, is that any of those nodes could die at any time. Right. And this is the sort of thing where like this is technically always true, you know, even when you're just running your code on your laptop, your laptop can die at any time. It's a, it's a physical machine, but this isn't such a concern when you have one computer, but when you're running on a thousand, it's almost guaranteed to happen once a day just because you've got so many machines there. And so we put the data into your container for you to process and then when you finish processing it, we check, we write it back out to object storage. And that's where once it's in object storage, that's when it's actually persisted within our architecture. Because nothing that's stored on a disk in a container, any of that stuff could disappear at any moment, is basically how we operate. This is also a great opportunity for me to talk to you about what the actual interface that your code gets to the Pachyderm data is. We really, really wanted to build a system that was going to be language agnostic. One of the things that really bugged me about the Hadoop ecosystem was that you sort of had to write in Java to really get the most comfortable semantics. Like you could kind of use Python, but it was always a little bit cludgy. And so when your code that you've put in a container boots up and because Pachyderm wants it to process some data, you will just find your data sitting on the local file system under a directory called pfs. And these are just totally normal files. You can open them with the system call open and you can read from them and write to them and stuff like that. And so this we thought was just the most natural interface that your code could possibly have. And users often have the experience when they've just written a jupyter notebook to process some stuff on their laptop. Normally they're just getting that data from local disk too. And so they have the experience when they're getting onto Packet or like, okay, like I'm going to need to learn the Pachyderm API. I'm going to need to like import Pachyderm into my Python code or something like that. Like no, you can just, you know, just use your normal OS system calls to open data and write data out and that's the entire system. That's all you need to do.

Yeah, so I have a follow up there and maybe there have been some updates that I'm not aware of. But I think one of the, the common kind of maybe struggles that I've seen people ask about is, you know, this is definitely fundamentally different than something like Hadoop or Spark where you have like some concept of data locality. Here you're kind of like putting data into the container and then taking it out, but it actually lives somewhere else. Are there concerns with that? Are there like, like trade offs? What is, what are the sort of trade offs that you're playing with there? Especially as you get into kind of larger data sets and that sort of thing?

Yeah, so there's absolutely trade offs, right? Because each time that means that the data needs to be downloaded from S3 written to local disk, which is normally faster than S3, so that doesn't really incur a penalty. And then it needs to be pushed back into S3. And so basically what you're trading off here is that this system could be more performant if it was entirely using hard drives, but it would be basically harder to, for, for admins to maintain. Right. Because the, the thing that people like about object storage is that it's just really dumb and simple. You've just got a bucket sitting there with all of the data in it. There's no like, which hard drive is this on? Like do we have all of the hard drives? Are they linked up to the right things and stuff like that? The reason that we chose this architecture as, as sort of our initial architecture is that this was a lot of the direction that we saw. We saw people basically making the same trade off in Hadoop, even though they didn't have to. So by far the most common Hadoop cluster that we see today, and this applies to Spark as well, is basically everything stored in object storage, almost always S3 and then MapReduce on top of that. And a lot of people are just bypassing actual HDFs at this point. We have been making over the last release and we're going to do a lot more of this in the upcoming 1.9 release. A lot of progress toward using hard drives to cache stuff. And so we're sort of going the other way that Hadoop went where they were first a hard drive only solution and then they started having like S3 as a way to like checkpoint stuff out to long term storage. And then eventually that started becoming the only way that people ran stuff. We're always going to have object storage as like the long term place that we checkpoint stuff out to. And then we're going to use hard drives on top as like a cache and that'll also allow us to use boatloads of memory as a cache too, similar to Tachyon if people want like really, really low latency stuff.

Cool.

Yeah. The times that I've interacted with Spark, I kind of like, I always defaulted to that S3 option anyway because it was hard for me to figure out other things. I don't know if that's just my own, you know, my own ignorance or whatever it is, but I definitely hear you on that on that front. But yeah, it's kind of like there's always trade offs, right? You don't get anything for free, but it's really kind of what you want. What do you want to optimize for?

Yep, it's always trade offs. And actually one of the things that we do a lot of is trying to counsel people to not worry as much about like performance on the margins in the early days because we've seen a lot of like infrastructure deployments and like data science projects that just get really bogged down and thinking like, well there's going to be this extra cost of data getting copied from S3 and getting back and stuff like that. And we always try to tell people like worry about these things if it's truly going to make it impossible for you to accomplish your goals, like if this absolutely needs to be a low latency system because you're doing like algorithmic trading or something like that. But in a lot of cases we feel like people get better results by just focusing on getting something that works. And that's I think exactly the trade off that you were making when you were setting up Spark is that like, yeah, if you really bang your head against the wall, you can figure out how to set up S3 on solid state drives, on AWS and it's going to be faster than what you're doing, what you're doing with S3. But if you consider the amount of time that you spent setting that up as like performance time until you actually get your results, you might actually get them much, much slower. There's a huge amount of value in just having infrastructure that you understand top to bottom. And that is simple.

So wanted to ask about that. We've kind of talked about a lot of different technologies, you know, in these, in these potential use cases. And I know that kind of getting back to teams and individual skills led teams where the skills were, you know, varied fairly widely. Some people like myself came from software engineering into the AI world and machine learning world and others came straight out of school and you know, with data science degrees and had not, had not done some of those. Do you ever find that there is any challenge or intimidation where people come out and they may know their data science but you know, they may not have even heard of Kubernetes or not be Familiar with containerization. I kind of wanted to call that out because, like, you know, me and you and Daniel are all incredibly familiar with containerization and kubernetes and such, but not everybody is kind of. How do you speak to that? Do you. Do you recommend a data engineer or infrastructure engineer get involved or what have you run into in real life?

Yeah, so that's definitely a challenge for us. And we really see the full gamut, and it's just very, very interesting. You see some people who, like, bill themselves as, like, look, I'm a. I'm a data science person. Like, I've never really done any serious software engineering. Like, I don't really keep up on this stuff. And then you sort of just sit them down and explain, like, all right, well, here's what Docker is like. Here's how you install and stuff. And they're like, oh, this basically seems to make sense. Like, I can get by here. And then there's some people for whom we. We do, like, education sessions and basically just try to teach people the basics of containers so that they can work with it. I would say that actually when we really have challenges, it has. It's less about software engineering expertise and probably more about DevOps expertise, to be honest. Like, a lot of the. The types of issues that we hit are just like, the permissioning on the Kubernetes cluster is wrong. And so when you go to deploy, like, your code, everything works until it starts trying to, like, talk to S3, and then, like, the network just doesn't work or something because, like, the bucket is rejecting it or something like that. And like, there's just a lot of DevOps complication in there. And so, you know, we always. We always sort of try to, like, keep our feet on the ground a little bit on this stuff because, you know, our whole goal with pachyderm was when I was at Airbnb is like, well, this data infrastructure is really hard, and my team is 25 people just keeping this darn thing running. And so what are all of the companies that don't have a team of 25 people to keep their data infrastructure running doing? And so we wanted to make something where you didn't need that team. Like a data scientist could just do it by themselves. And I think we're closer. But, you know, then when we go into companies and talk to them, they're like, well, we've got, like, one person working on this full time, and, you know, they're feeling like they have to do a lot of DevOps to keep the pachyderm cluster up and running. I sort of realized, like, okay, you know, we haven't, we've made an improvement here. We haven't just magically eliminated this. You know, we haven't gone from you need 25 DevOps people to keep big infrastructure running to you need 0 DevOps people to do it. And so we're trying to make that better in every release. We're trying to make that as easy as possible. And one of the big steps forward on that will be having our own hosted solutions so people don't have to deploy everything on their own cloud just to try it out. Short answer is that's definitely a challenge, is that there's a bit of an infrastructure leap that needs to be made, which can be uncomfortable for a lot of people that I think could ultimately benefit from the feature set of Pachyderm. It's just they can't quite get the activation energy.

So I was wondering, is there anything else, you know, another question that you commonly find is people have existing infrastructure in place. They might be a Hadoop shop, a Spark shop, or one of several other technologies. You know, they might have big databases like Cassandra. What are you trying to replace and how are you trying to fit in? I know we talked about the data locality issue, but are there any other big considerations that you would say is why you should go Pachyderm versus what they already have in house?

Yeah, I mean, I would say the things we're trying to replace are sort of HDFS and then the computation layers on top of that. So, like, MapReduce is the common one, but like Hive and Spark and stuff like that. We're also trying to speak to. Those are the main things that we're trying to replace. We constantly have the challenge of, with people who have existing data infrastructure and want us to sort of fit into that. Well, and that's always a bit of a, a back and forth because some things can work really well in Pachyderm because you can just, you have the flexibility of a container and so you could put whatever you want in there. So, you know, people will have containers that include code so that they can go and talk to HBase somewhere else in the cluster. Right. And so then you have sort of a natural like shim to put between your existing infrastructure and pachyderm, which is the container code, which is totally flexible. It doesn't work beautifully for everything. Right. Like what you, you wind up doing with like Spark or something is you wind up having, like, here's Your data, it's stored in Pachyderm. Now you boot up a job and you want to talk to Spark. So now I need to push all of this data into Spark or somewhere where I can access it or something like that. So we're sort of constantly trying to figure out how to make these integrations better. But the users that always excite us the most are the people who, who basically come in and say like, we don't want to go down the Hadoop route. Like, we know that there is a lot of just pain required to get a working Hadoop cluster and to get stuff functional on it. And so we want to try something different and just build from on Pachyderm from scratch. And so long term for our company, we're focused on how can we make things really good for people who just see the packet or vision and commit to it from scratch. Because those are, you know, if we're successful in 10 years, then those are going to be the people that have really made the company successful. And the sort of the integrations will help us along the way to onboard more people, but it's really going to depend on that core use case.

Yeah, so the team that I'm working on now, the organization is pretty big, but it's kind of on this project that I'm working on. It's like myself who has some type of data science background and then another guy who is somewhat technical, but he's a linguist. And so our ability to spin up a working Hadoop infrastructure is probably less than 0% probability. And so I mean, even just like if there's one thing I could say to listeners, like even if you just get to where you can use containers themselves is like a huge benefit also to reproducibility in the space of machine learning and AI, which is awesome. So I kind of wanted to follow up. You've already mentioned, JD that pachyderm, @ least what we've talked about up to this point, is free. But I also know you're a company, right. And I should give you some congratulations because you just kind of hit a big accomplishment, isn't that.

Right? Yeah. And thank you for the congratulations. We just raised a Series A, which means that we have a ton more funding to basically pursue our vision for data science infrastructure. And it also means that you can commit to Pachyderm as your infrastructure with a lot more peace of mind now because you know, the company is going to be around for quite a ways to come that also sort of leads, as you said, we are A company which means that we need a way to make money and that for that we have an enterprise product. So let me just sort of tell you what's in that that you won't find in the enterprise. We try to really make it so that our open source product contains everything that's going to be really useful to sort of individuals and people who just get some, some data science done, but they're not running within a gigantic organization where they have all of those concerns. So the types of things that go into that enterprise product are the permissioning system. And so that's, you know, the ability to say like this data right here is owned by Dan, this data right here is owned by jd this data right here is owned by Steve, things like that. And make sure that nobody is getting data that they don't have access to. And what's cool, and what we think is a very crucial feature for these types of systems is that it's informed by our provenance model. Right. This is a big problem that you'll run into in big data organizations is that it's very easy to have some data that nobody's allowed to see that then gets turned into a model or some sort of an aggregation or something like that that everyone's allowed to see that is accidentally leaking the data that went into it. And so we have our Providence tracking system inform the permissioning system. So if you don't have access to the provenance of data, then by default you don't have access to the data itself because it might contain that information that you're not allowed to see. Other things that go into the enterprise product are like a sort of wizard UI builder for building new pipelines and things like that and visualizing how they're working and the ability to sort of track and really optimize your pipeline, see like where they're spending all of their time and like squeeze every last little bit of performance out of your hardware. The other main thing that we sell is basically just support and our time and you know, the, the ability to like talk to us and have us prioritize features and stuff like that, which is, you know, every open source project does.

That. Yeah, it's really interesting. It's. I always love to hear like different people's perspectives on their, their open source models as well. I was just talking to someone the other day, a friend who's starting a new business and like considering how they should approach open source, but yet also be a company and like and survive. So I think there is definitely people out there that are interested in that question. So I appreciate you sharing.

That. Yeah, absolutely. And you know, it's tricky and it's very imperfect because I really think that this is a system that really should exist. There's a lot of need for a system like this. It basically has to be open source for it to actually fill that need. In my mind, I just couldn't see a proprietary system becoming like the standard data infrastructure layer. But it's very, very hard to get the funding to work when you're open source. You know, it's this huge asset because people can so easily try your product and you get so much adoption and stuff like that. But it really anchors people of just like an unwillingness to pay for, for software when it's open source. And so you always sort of need to cross that threshold. And one of the things that we're looking to do in the future now that we've raised more money, is basically build the hosted version of our software. Because that just sort of totally, it totally changes the value proposition. But it also, I think, has some sort of psychological effects on people wherein, like, nobody would ever pay for git. But the idea that you're going to pay 7 bucks a month to have like private repos on GitHub or something like that is just totally palatable to.

People. I think that's a fantastic idea. I love, I love the hosted idea. I know that when Daniel first introduced me to Pachyderm a while back and I was kind of initially learning the fact that coming from the software engineering world that it was built on containerization and Kubernetes was a huge plus for me, if I recall correctly, a lot of it's in go, which I thought was pretty, pretty amazing, as is Docker and Kubernetes. I guess if you're just hearing about it and you've kind of come away from this episode today and you want to learn more about it and maybe want to dive in, get your hands dirty and figure out if it's right for your organization. How do people get started with.

That? Yeah, so we've got a bunch of tutorials and like quick start guides online. And so, you know, if you want to just sit down with a guide and start hacking away, then, then that's the way to do it. We also have a very active user Slack channel, where all of our engineers and everyone on the team is just always hanging out and ready to ask questions. And, you know, those questions range from like, I hit this error, what do I do? And you know, we just we just give you a simple response if it's simple. Hopefully it's simple. And to people also asking us, you know, I'm looking at Pachyderm for a new project. Talk to me about the feature set. You know, talk to me about how you think this could be helpful here and just like, like talking to us. And so I think that's, that's really the best way if, if you want someone to talk about, to talk to, about stuff is just stop by the Slack.

Channel. Awesome. Well, thank you so much for taking time to Talk with us, J.D. of course we'll put the links to like the tutorials and the docs and the Slack channel and all of that in our show notes. So, so go check those out. But it's been awesome to hear from you and really excited to hear about the, the progress with Pachyderm and all the good things, things you're.

Doing. Yeah, thanks so much for having me, man. I love appearing on.

Podcasts. All right, well, look forward to seeing great things from Pachyderm. Thanks.

Again. Thanks for coming on the.

Show. All right, thank you for tuning into this episode of Practically High. If you enjoyed this show, do us a favor. Go on itunes, give us a rating. Go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you got to do. Share the show with a friend. If you enjoy enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more@fastly.com and we catch our errors before our users do here at Change. All because of rollbar. Check them out at rollbar.com changelog and we're hosted on linode cloud servers. Head to linode.com changelog check them out. Support this show this episode is hosted by Daniel Whitenack and Chris Benson. Editing is done by Tim Smith. The music is by Breakmaster Cylinder. And you can find more shows just like this@change law.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next.

Week. I'm Nick Niese. This is K. Ball. And I'm Rachel.

White. We're panelists on JSparty, a community celebration of JavaScript and the.

Web. Every Thursday at noon central, a.

Few of us get together and chat.

About JavaScript node and topics ranging from practical accessibility to weird web.

APIs. You could just eval the text that you're given and.

Then. And.

That'S. I think that's basically what it's.

Doing. What could go wrong? Yeah.

Exactly. This is not legal advice to eval. Text as it comes in. Join us live on Thursdays at noon.

Central. Listen and slack with us in.

Real time or wait for the recording to.

Hit. New episodes come out each.

Friday. Find the show@changelog.com jsparty or wherever you listen to.