AgileData #21 - The magic of serverless

Welcome to the Agile Data Podcast, where Shane and Nigel discuss the techniques they use to bring an agile way of working to the data world in a simply magical way. Welcome to the Agile Data Podcast.

I'm Shane Gibson, and I'm Nigel Vining.

And today, Nigel and I thought we'd have a bit of a chat again, another kind of engineering-focused chat, but this time we want to talk about this thing called serverless, which is something that we use a lot on the cloud platform that we leverage. For me, it's a really, really funny word. In fact, a number of years ago, I was very disparaging of the term serverless because there actually is servers involved, right, Nigel?

Yeah, it is a misnomer. There's a server under the covers. It's just what the definition of serverless actually means to you, I guess.

Yeah, so why don't we start off with trying to define how we would determine if something was serverless or not.

Sure, so my definition of how we use it is, uh, you don't own the server, you don't have to manage the server, you basically use that server on a second-by-second or minute-by-minute basis, and you pay for that fraction of the time something is actually executing. 99% of the time you're not doing anything with that server, so we call it serverless.

Yeah, so for me, I like that definition. I like the definition of when I ask it to do something, I get charged, and when I don't ask it to do something, I don't get charged. So for me, that is one of the tenets of a serverless piece of technology and one of the benefits it gives me.

Yeah, I agree. I like it for the simplicity that you don't have to worry about all the overhead that goes with a server in the traditional sense of the word, which is data centers, networking, patching, upgrading, basically all the downsides of running your own server. You don't have any of that with serverless. You just ask it to do something, it does it, it always works, and then it stops.

So if we compare, you know, something that's serverless versus something that's based on a container, You know, for a container, I don't own the server, I don't own the network, I don't own the disk effectively, I don't own the infrastructure. But in my definition, it's not serverless because I have to turn it on and I have to pay while it's running, whether I'm actually using it or not, it's sitting there charging me, right?

Ah, yes. Some of the products on most cloud platforms are badged as serverless, but you are paying an hourly rate effectively because that server is always available to run your container. It never goes to zero, is what we'd say under the covers. There's always one of it running.

So it's a virtual server then, right?

Yeah, yep, that's a good definition.

Okay, so why don't we run through all the moving parts we use and then try and have an idea of whether we think they're serverless and what the value of them is. So So let's start with the big one, BigQuery, where we store a lot of really cool data. So what do you reckon? Serverless, virtualized server, physical server?

BigQuery is an interesting one because there's two things going on there. There's the compute resource, which I'm going to say is effectively serverless because if it's not doing any work, you're not being charged. But behind that, you've got your stored data, which is being stored and you are being charged while it's been stored but not being used. So BigQuery is a bit of a hybrid in my view.

And that's when we use a charging mechanism against it, right? If I use a metric of how am I billed to define whether it's serverless or a virtual server, then BigQuery really fits in the middle.

Yeah, it's got a split cost. You're paying to store over the long term. Obviously, as we talked about a couple of weeks ago, if you're not accessing your data all the time, it goes off to lower tiers of storage and it's cheaper. But if you're not running queries, BigQuery is not charging you anything.

So, so one of the other ways to look at it is with a virtual server, I've got to install and maintain the software, right? I mean, I may use some form of template or image or some magical provisioning thing that grabs it off a marketplace and that stuff. But really what's held within that virtual container is mine. I control when it's upgraded and when it's not. I can control how it's configured at any level and how it's not. I have access to the operating system 9 times out of 10, right? Whereas if we look at BigQuery, we have none of that, right? We don't control when new features are added or when it's patched. We don't have any access to the underlying operating system or the infrastructure that it runs on. You know, we don't go to a marketplace, we just tick a box or use code in our case to say we want to access one. So is that kind of the, the second way we can look at it? Is one, how we're charged, and B, what we control?

Uh, yep, that's a good one. That naturally flows on to— it's got me thinking about, um, Cloud Functions, which we make use of, and that's exactly that case. We pay per second of compute time, but we don't control the, I guess, the architecture or OS. We choose what type of architecture we want to run our code on up front. For example, we'll say I want to run a Python Cloud Function, and all I provide is literally the code I need to run. Python is pre-installed, it's patched, it's good to go. We don't control that. We've just asked for a Python environment and we throw stuff at it. and we get charged per second to run it.

And so if we look under the covers of Cloud Functions, because we spent quite a bit of time looking under the covers as much as we can with Cloud Functions, really it operates differently to BigQuery though, right? So, you know, the perception I get is BigQuery has a whole lot of infrastructure just sitting there that everybody kind of hits. and when we want to run something, it runs it for us. It allocates us some resources off that big cluster-y thing. But when we look at Cloud Functions, we can kind of see that they're stopping and starting virtual servers on a regular basis every time we want to run something. So architecturally, they're running, you know, one's more of a, you know, a clustered shared environment, and the other one's more of an on-demand virtual server environment. But from our point of view, we pay for what we use and we don't control any of the underlying infrastructure or operating system for either of those services, right?

Yeah, which is actually nice because effectively there's nothing to worry about. We throw code at it, it runs. If we throw two bits of code at it, it starts 2 instances and runs them. If we throw 100, it still does the same behavior. It is a consistent behavior. You can run 1 or 100 Cloud Functions simultaneously. You do not have to worry about it. It's, you know, it's a, I'm going to just say it just works.

So let's look at the other end of the extreme for us. So, you know, we do documentation as code. So we, We write small bits of code that is effectively our documentation. Whenever I check it into our repo, some magic happens and our documentation site gets rebuilt with that latest version of all that documentation and code. But that's not a serverless architecture that we're leveraging for that puppy, is it?

Sort of. It's midway. I guess it's a, the serverless architecture is a continuum from the small ones that we pay per second and we have no control over to the ones where you talked about containers and VMs before. So for our documentation, we have an image which contains our, I'm just going to call it our documentation engine, which is the Sphinx open source library. We have an image of Sphinx that's customized to how we want to build our documentation. And effectively, when we do a build, it mounts that container, runs it for a period of time while it builds our documentation, then it shuts it down. So it is serverless in some sense of the word, but it's different in that we're effectively mounting our own custom image. So we get to choose what's installed on that image and what's running, but we're still just paying for the amount of time that that image is mounted and running, and then it basically shuts down again, and it's nothing, but it's upper tier from a Cloud Function, but it's not a full, you know, constantly running server. We mount it, we use it, we drop the image, and it's done.

Okay, but my understanding is that serverless-ish behavior is something you've engineered, right?

I guess so. We needed a level of customization that didn't come with a Cloud Function because we needed to customize the, I guess, the OS effectively and install some additional products that weren't available in a Cloud Function. So we built our own virtual image that we run for a period of time and then shut down. It happens automagically for us. It's just another level of serverless type behavior.

Yeah, so, so it's not a service or a serverless feature we get from Google. What you're doing is leveraging some of the Google services to provide a capability that behaves in a serverless pattern. Right, you copied the pattern, but you had to build it. And so if I go back to my two checks, do we pay for when it's not running? No, because you've defined, used a pattern that says when it's not running, turn it off. Do we control the operating system and that container itself, and are we responsible for patching and maintaining that container? And the answer is yes, right? So, so we get one of the benefits of a serverless pattern, and the other one, we, we wear the cost and effort of maintaining that.

That's exactly right. If we want to update the Documentation Builder, install more features, or patch it to a newer version, we basically have to start up that image, update it, and then repersist it so we can use it next time it's called upon.

Cool. So what else we got in the technology stack that we can have a look at? What about the cloud repo, where we store our magic code?

Yeah, I was going to say the next on top of the cloud repo, we make use of the cloud build functionality, which I guess is a form of serverless products. What we do then is by using commit triggers on our repos when new code is committed into specific repositories. What we basically say is if the master branch of, say, I don't know, documentation is updated, then I want you to use Cloud Build, which is a serverless feature, to run a whole lot of steps for us, deploy some code, check it, and then shut down again. So Cloud Build's a little bit special because we effectively can perform a whole set of instructions. So it might say get the latest code that's been committed, do this to it, do that to it, copy it here, copy it there, and then send me a message to say you've done that. So again, Cloud Build is charging us per second that it's running to perform our instructions. So we typically use it to pull the code out of the repo, do some functions on it, and then possibly copy the output into a Cloud Storage bucket and then shut itself down again.

Cool. So, so do you tell Cloud Build to start and stop, or does it just wait for you to call it and then when it's finished its job, it shuts itself down?

So it's triggered by something happening, in this case a commit to a repository. A user can invoke it directly off the command line. I can say run this Cloud Build script while we're developing it. It works in exactly the same way. It's, it's, think of it as a recipe. It's got a list of instructions doing it. When it started, Cloud Build reads through the list, does all the instructions, and then shuts down.

So, so from a serverless costing point of view, Google Cloud takes care of that serverless behavior for us.

That is exactly right. It's running on our server in the background. It's not ours. We don't maintain it. It just runs a set of instructions.

Cool. So that's the second question is we don't maintain that operating system. We don't patch it. So it's a true serverless offering from Google rather than something where you've applied the serverless pattern.

Exactly. Cool.

All right. And Cloud Repo, same thing, right? We just check code in. And when it needs to receive that request, it does something and receives it. And if we don't talk to it for a while, it doesn't care and we never, and neither do we. Well, we care because we're not checking our coding, which means we're either not working or we're doing bad, bad things from a day ops point of view. But as a serverless component, it doesn't care. Yeah.

Exactly.

Cool. So what about the code that we use to go and hit some click data from some of the Data Factory? Let's say, you know, when we go and hit Shopify and grab the data out of that puppy, you know, which of the patterns are we using for that one?

We actually use Cloud Build and a container for that one. We use a container because the container has the particular software installed on it that we use to talk to those APIs. So we've installed that library on the container, and when required, we effectively run that container, uh, and then we shut it down again.

So we've applied a serverless pattern. Yes, to something that's more of a container model, right? Because we still control the operating system, we control that that container image, we control those things.

That's correct, because we've installed a number of pieces of open source software in that container that have the libraries for talking to the APIs in question. So we maintain that, we just use the serverless pattern to run that container for us and get charged per second while it's running and then shut it down again.

So why wouldn't we just use the serverless Pacman out of Google Cloud instead of having to apply our own patent for this one?

We needed to install the software we wanted to run somewhere. So our options are we can have something that's literally a real server that's running 24/7 and we're paying for it the whole time, but we only want to execute those libraries once a day, maybe.

I think the other thing was when we did it from memory, or when you did it from memory, the open-source software we're using required certain permissions or behavior from the server or container it was running in. So when we tried to deploy it into the serverless services, you know, there were— it effectively didn't run properly, right? There were things that relied on that serverless capability didn't allow us to do because we don't own the operating system, we don't own the underlying disk, we don't own a whole lot of moving parts that actually that particular piece of software assumed you had access to and was designed to actually leverage. And therefore, you know, computer said no, right?

Actually, you've just jogged my memory. At the time when we started this journey, the Cloud Functions that we would have used for everything had a timeout of 3 minutes on them. Which is why we got bumped up the additional serverless ones, because at the time, the, the smallest tier of serverless was 3 minutes, the next tier was 9 minutes, and then the next tier up was as long as you like. 3 minutes wasn't long enough to basically set up the environment, run the APIs to pull data, and bring it all down within that 3-minute period, which is why we actually had to deploy a container which didn't have a time restraint. But since Google was very helpfully 3 times in the last 18 months bumped up those time limits. Cloud Functions can actually run for 9 minutes now from memory, which is potentially long enough to do a data extract using an API.

Yeah, so there is still a risk though when you use a serverless feature that there will be some type of access or capability that you would normally have be able to leverage on a container, on a physical machine or a virtual server that that serverless thing doesn't allow you to do, right? And so therefore you have to regress back to applying some form of serverless pattern to that container-based pattern, right?

And those restrictions are probably likely around CPU, memory, and disk because using serverless, the true serverless, you're effectively using a a preset environment that's got, say, 2 CPUs, it's got X gig of memory, and you've got access to X gig of temp storage, I'm going to say. So if your requirements are for something that's very memory intensive, and I'm just going to say, for example, a Cloud Function may only give you 4 gig of memory, but really you're going to need twice that much because you're going to do something chunky, that's not going to work for you. Or if you need to physically store a whole lot of data on a local disk, again, that's potentially not going to work for you, which is where you go to your own pre-built image because you can say, I need a container that's going to have access to plenty of disk or plenty of memory, and that might be a guiding choice.

So, so really it's quite confusing, right? Because you can get something out of the box that's serverless. You can get something out of the box that behaves serverless because you only pay for what you use and you don't manage the operating system. But you was talking about number of CPUs and amount of memory, which to me makes me think of a server. And then you can take virtual containers and make them behave using a serverless pattern where they only run when you need them to run, so you only pay that way. and you're effectively using some form of template to manage the operating system. So, although you're still having the pain of managing it, it's kind of magical when you use it. So really, you know, there's a lot of things to think about when you talk about, is it serverless or is it not?

Yeah, it becomes technical after a while, and I think what makes it more challenging is there's multiple products on a spectrum and they all slightly overlap. So the smallest, you know, offering may actually cover all use cases well up until the middle offering. And, you know, the middle offering may do all your small use cases and then into your large use cases. So it's no— there's no hard and fast rule that says this is my use case ,so I automatically use the serverless function because to be honest, all of them might satisfy that. Then you've got to make a second choice around how fast will it be when it runs. So I might choose an architecture that's going to be quicker or more performant with less latency when it starts up and shuts down, because some of the serverless comes with a small overhead because you're effectively, you've made a request, it's now got to effectively start itself up. To deliver your service and shut itself down. It may only be a second or a fraction of a second or 2 seconds. That may be an important consideration for you.

So I think, you know, for us, it shows the value of our guiding principles from an architecture point of view. And by that, I mean, you know, at the beginning, when we need to do something new, we will always look for a serverless component that enables us to only pay when it runs and not have to wear the cost and effort of maintaining an operating system or patching or managing that container. And if we find one of those and it meets our needs for now, that's what we use, right? And we know that we can always refactor that service, that Pac-Man, later on to something that we have more control over but takes more effort and therefore more costs on our side to maintain. So the guiding principle of serverless first, but not serverless over everything is really, really important, right?

Agreed. Yep, we always look for the serverless, the smallest serverless option, and then we just go up a level if it doesn't quite do it, and up another level if we have to. But serverless-first principle.

Yep. And then always ready to refactor because we know that over time Google, in our cases Google Cloud, adding more and more serverless features. Or removing some of the barriers of the serverless stuff that we were using to mean that we can actually refactor stuff where we've gone for more control for a reason back to being able to leverage the value of that service. Absolutely. I struggle with the word serverless service. I don't know why, I think it's an alliteration, isn't it?

A misnomer.

Yeah, it's, It's just a bad sentence when I say it, or a couple of words.

So the future is serverless, I think, is my takeaway.

Well, actually, I think the future is biometric computing. So the real question then is when we move from silicon to organism-based compute, is that a server or is it a host? That'll be a really interesting interesting distinction when that comes out in whatever years it is. But I think for now, I think we've kind of talked about serverless and we'll probably put that one in the can.

Done, thanks Shane.

And that, Data Magicians, was another Agile Data Podcast from Maju and Shane. If you want to learn more about how you can apply agile ways of working to your data, head over to agiledata.io.