What’s it like to be a Site Reliability Engineer with Laura Stone

Five, four, three, two, one.

Drop dead.

Welcome to the Test Skill Performance Podcast, where we get together to learn more about performance testing and site reliability with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance Podcast. The full name is the Test Guild Performance and Site Reliability Podcast, and that's why I'm so excited about today's episode. So if you want to learn more about what it takes to become a site reliability engineer, you're in for a treat. We have Laura Stone joining the Guild, who is a senior site reliability engineer at Klaviyo, who will share some of her experience working full time as an sre. So listen in and discover Laura's favorite tools and tips for success as we go through a day in the life of a of a site reliability engineer. Check it out.

Face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey, Laura, welcome to the Guild.

Hey, Joe, thanks for having me.

It's awesome to have you on the show. I met you at Perf Guild last year. Really, you have a lot of knowledge and a lot of different things. So today I'd like to focus a little bit on site reliability engineering and mostly for beginners, and also your experience around DevOps. And you also have a bunch of courses on LinkedIn, so I thought we'd dive into a bunch of different things. But Laura, before we get into it, could you just tell us a bit more about yourself?

Absolutely. So my name is Laura Stone. Right now I am a site reliability engineer at a company called Kladio, which is in the Boston area and it focuses on marketing automation. I'm on right now a very small team of three. We are responsible for all of the supporting services at the company in engineering. What does that mean? We have a bunch of product engineering teams and they're responsible for delivering the product to our users, writing features, owning different parts of the domain of the Klaviyo application. Then my team is responsible for building up a platform that the product engineers use and then giving them a bunch of supporting services, tools, guidance, that sort of thing. So that they can go faster so that they can do their work reliably and so that the things that they make their architectures ultimately can scale. That's what I do at a very high level.

You sit outside of an Agile team, a Scrum team, and you have support for all these other teams. Is that how it normally works?

Yeah, the thing that's really cool. And there's actually a website that I'll plug really quickly called DevOps topologies. If you just type that into Google, it'll find the right place for you. But there are a lot of different models for having teams that do operations work that support product engineering teams. And so the way that my company has chosen to do this is we are called the Site Reliability Engineering Team. Right now we are a separate SCRUM team. You know, we go through all the same rituals in terms of Scrum as product engineers. We have standups, we have grooming, we have retros, we have planning. And then what we do is we, when needed, embed ourselves briefly on product engineering teams to sort of help them either get off the ground or solve a particular problem that relates to either reliability, scalability, security, that sort of thing. So again, the way that it works on my team is we are our own separate team. We are separate from product engineering. The only time we'll ever really embed is on a particular problem for a specified amount of time. But that's not necessarily how other companies do it. And that way isn't necessarily more successful than other ways. It's just what works for our company.

So what's an example of a supporting service? How do you get work and then how do you divvy it up? How does your group work on things? Know what to work on?

Yeah. So the supporting services that we provide are, number one, we have a homegrown deployments tool that we use for our monolith. So our architecture is kind of bifurcated at the moment. The majority of our platform is written in this Django monolith and that is in Amazon, it's on EC2. And there's a lot of automation that we have built up around that to make it work.

We have a deployments tool that is the responsibility of my team, that helps the product engineers deploy their code. So that's an example of a supporting service. Another supporting service that we provide is a metrics aggregation platform. So going to plug a course really quickly. If you're not familiar with statsd, Graphite and Grafana, they're an open source metrics aggregation platform that you can build and use to get insight into your application's performance. I have a LinkedIn learning course on that. You can look it up Graphite and Grafana. Visualizing application performance. That's another service that we provide to the product engineers. They can essentially any arbitrary thing that they want to be tracking, they can track it in the application code, it will aggregate in our platform and then they can query the platform through a front end, make graphs and stuff like that so that they can visualize how their part of the application is performing in that one area. That's another example of a supporting service that we provide. We also provide centralized blogging. We also have a Kubernetes platform that we are in charge of. That's the sort of stuff that we do day to day in terms of the services, in terms of how work gets into our backlog, there's two ways. Number one, there is us ourselves working with our business stakeholders to write user stories essentially as part of features and epics around those services. Or maybe we're thinking about a new service. For example, we want to have a secrets management solution that product engineers can use.

That's one way. The other way is we also have a Slack channel for incoming requests from the product engineers and as they need them. These are the more sort of like operational tasks that need to get done. Someone needs their permissions changed in Amazon, someone new needs to get onboarded, someone needs a secret value change and they don't have permission to do that, that sort of thing. So we have a Slack channel and we have a separate sort of Azure board. We don't use jira. We use a different tool called Target Process, but essentially it's jira. So we have a separate JIRA board where they can file that incoming request and then we have, you can think about it as an on call rotation, but it's essentially like a bug triage rotation. Who for one week that individual is responsible for just owning that board and as requests come in, doing them, prioritizing them and then again just like trying to get through as much of that as possible. So that's how work gets in our queue and how we get it done. Again. We have one week sprints. So we take from our backlog, from our features that we're working on and then we put them in the sprint and then get them done as well as our what we call the help SRE board.

I should have asked this earlier, what is site reliability engineering? Because a lot of the activity you described to me sounds like DevOps. So for someone that's new to site reliability engineering. How would you explain it? What's the definition to you?

Yeah, so the way I like to explain it is you can read it probably from the site reliability engineering book from Google. But, but it's when you treat operations as a software engineering problem. So what does that mean? That means that we perform a lot of operational tasks. And what I mean by operations isn't like business operations, but again engineering operations. Things that traditionally have been done manually, but instead of thinking about them as just these manual things that we do, thinking, okay, yeah, there's going to be some amount of manual work that we're going to have to do, but what automated solutions can we bring to the table so that we don't have to keep doing that manual work? How can we provide either self servicing tools or the right guidance to engineers such that they kind of need us a little bit less? And I guess the idea is how can you automate yourself out of a job? Right. In some ways I think this is really similar to what people might refer to as DevOps. I personally think that DevOps is more of a philosophy and it's more of an organizational inclination and less of a sort of organizational function. And I think of site reliability engineering as a function within an engineering organization that is focused on operations, but it's not the traditional operations that you saw 20, 30 years ago. It is a new form of operations where site reliability engineers need to know how to write code, they need to know how to write applications, and at the same time they need to have that operational expertise in things like proxies, in things like databases, in things like systems internals for your operating system.

Wow. So Laura, I mean, I'll be honest, it sounds like you need to know a lot of things. So how do you not find someone with all these skills? It seems like a unicorn type position. Yeah.

I think that in general, number one, site reliability engineering is really hard to hire for because of exactly what you just talked about. And the other thing that I think makes it hard to hire for is there's no predetermined path in terms of getting into this field. There's no like, oh, well, if you go to school and you get a computer science degree, you'll know how to do everything to get this job done. Where for product engineering I think a lot of people find that sufficient. But for site reliability engineering you could go to school and know all of that and still have these huge gaping knowledge holes. And I haven't seen a lot of companies solving that problem really well. But what I will say is there are ways to learn those skills outside of school and there are a lot of really good resources online for okay, I want to learn how to set up a mail server. There's resources out there to teach you how to do that. Oh, okay. I want to learn about databases. There's lots of resources about that. And so I think that for people who are self motivated and who are willing to show that they can put in the hard work to learn what site reliability engineering is and the different parts of the domain, I think that they'll find there's a lot of hunger for folks who don't necessarily know everything but are willing to learn it. And I know that's true at my company. I know that we will take folks who maybe don't have all of the domain expertise, but they have some experience with Linux, they have some background in software engineering and they're willing to give it a shot.

How much programming is there involved? A lot of times there's different types of programming. I think there's actually programming for enterprise application, then there's fun. Programming to me is when I use Python to get little activities done to automate things for me. What kind of program programming knowledge does someone need to have in order to be a set reliability engineer?

I would say that it depends on the organization because again, I think.

Particularly when you're talking about DevOps, people have very different meanings around what that is. And so I think in a lot of organizations maybe just having some scripting experience is sufficient. I know that at my organization we expect site reliability engineers to, if they don't have explicit software engineering experience, to at least show the ability to write application code. To give you an example of what I mean by that, everybody that goes through the Klaviyo interview process, unless you have some equivalent, is asked to do a take home exercise. And that take home exercise is hey, please write a small web application where you can enter in your email address, you can put in your city and then write a command on the back end that will, for all of the email addresses that have signed up for all the cities, send them a code if it's nice weather or not. We expect anyone, even if you're coming onto the site reliability engineering team, to be able to at the very least write a small web application. Again, I know some companies don't require that, but we do. And I think that's just because the way that we think about site reliability engineering is that we are providing sort of our internal applications for our product engineers who are our users, they're our customers. And so we want to be able to do that as much as possible.

Awesome. Well, that's good to know. I'll be honest, I'm old. I haven't been on a job interview in 12 years. But my specialty is automation testing. I don't even think I'd be able to pass an interview now because a lot of them are coding specific like hardcore, so sounds like it's similar with site reliability engineering for some organizations as well. If you're going on an interview, you should hone up on your programming chops is what it sounds like.

Yeah. And the thing that I like about our interview process for that is you do get to do it at home, so it's not like we're breathing down your neck watching you do it. And again, that puts the emphasis on maybe you never wrote a web application before, but you know how to find the documentation and you know how to look up the things that you need to do to be able to do it.

You did mention some resources that people get up to speed. I know you have two courses on LinkedIn and you didn't mention one already, the graphite and Grafana Visualizing application Performance. So if someone's getting to say reliability engineering, I know dashboarding is big everywhere. Is there a reason why you chose Grafana over another tool like Elk Kibana or something like that? And also follow up question to that is what type of performance metrics do you use to help you visualize an application's performance?

So the reason that we chose Graphite and Grafana just at my company, number one, was because it was what was already there when we joined. So I know that the CEO is the technical co founder and I believe he was the one that originally set it up. So when I joined it was what was already there. But I think the other part of that is choosing open source over paying for something like Splunk was a decision that just in terms of the organization's values felt right now in terms of using something like Graphite and Grafana versus something like Elk, I think typically Elk stacks are used more for centralized logging and less for for just like raw metrics aggregation, although you probably could use them for the same things. I'm not super familiar in particular with Kibana's pros and cons to be able to say, but we have found like we definitely hit some scaling limitations at first with graph writing Grafana, but then by using some of the tools that I mentioned in the course that I did to be able to get beyond the basic graphite stack. We were able to get the scale that we needed and it was sort of like, why change the whole world if we can get the scale that we need? I will say.

For our Kubernetes small platform that we have going, we are using Prometheus, which is a little bit different. But there are also scaling problems that eventually will hit with Prometheus and then we'll have to see how we get around those. So it's not so much about any one tool. It's like whatever works for your organization. I would say go for it, but for us, we haven't hit any problems with graphite and Grafana that we couldn't overcome. So there was no real reason to switch.

Awesome. What kind of data are you visualizing to help you figure out the application performance? Are there any key metrics you usually use to see how healthy your application is?

Yeah, that's a good question. It depends on, number one, what the component is within the application, and number two, you know what, what are the business, what's the business value that you're trying to provide? Right. I think an easy, an easy answer could be, well, you know, there's always the big four, right? There's disk space, there's cpu, there's network, and whatever the other one is, I'm forgetting. I could tell you look for those things, but frankly, like if you're looking at Redis, it's gonna, you're gonna care more about CPU and memory in particular CPU in terms of the number of cores you have. Because Redis is single threaded versus caring more about how is the business doing. I would use something like STATSD or Prometheus, the Prometheus client, to be able to add arbitrary metrics. And I really can't tell you what those arbitrary metrics are for your organization. I can tell you for Klaviyo, we care a lot about metrics ingestion. So there's sort of two sides to Klaviyo. There is sending things out and then receiving things in. So we care a lot about the throughput of how many emails can we send out per second. And then again, we care a lot about the queue depth of those emails, how many emails are queued up and what is the delay on being able to get those emails out? Because we have some agreements with our customers about how quickly we ought to be able to send out their email. And then similarly, we have a sort of informal agreement with customers where we want to try to keep their data as real time as possible. And so then on the incoming side of things, it is, how quickly are we ingesting events from which sources? What's the delay? So again, I would say we care a lot about throughput and we care a lot about time to process for our business metrics.

Laura, I guess another question is, does your group do the triaging and then it's the Sprint team's responsibility to fix or do the performance tuning? Or is it like you actually have to fix it as well, not only identify it but also address it so.

We don't actually do the triaging of issues as they come in? Product engineering teams own their domains of the application, from soup to nuts, nuts to bolts, whatever the term is. If there is an issue with their part of the platform, they are the ones that get paged and they are the ones that are responsible for triaging and fixing. They will pull us in as needed if there is some expertise that we have that they need from us. For example, we have several databases that are utilized across our monolith. What actually happens is if we just let all of the different workers connect to those databases, it's actually too Many connections for MySQL to be able to handle. What that causes is a lot of problems at the application layer because the workers that are trying to talk to the database can't. What we actually do for those databases is, is we introduce a proxy tier that holds the connections from all of the workers and then only opens connections to the database as it's needed. That way we don't experience on the application side any connection problems to the database. But when problems start to happen with that proxy layer, once that product development team has seen that, they often don't have the expertise in it. So then they come to us and ask us to help them troubleshoot what's going on there. That's a good example of us coming in and helping. But in general, the application team, the product development team, they own.

Their entire part of the application.

So, Laura, how much if any, performance testing engagements do you participate in? For example, usually it's easy to create a performance test, but I found that finding the right people that know how to monitor the network and the database is kind of hard. So are you engaged at all in any sort of performance testing activities as like doing the monitoring piece or actually putting on a load onto your system?

Yeah. So something important to know about Klaviyo is that again, it's a marketing automation tool. And so a lot of people use that for email marketing. And what that means is that Black Friday and Cyber Monday are our biggest days of the year. And we scale probably 10, 10 plus X in terms of the number of emails that we need to send and the number of events that we need to ingest to be able to be successful on those two days and again the weekend in between it. And so we undergo extensive load testing and I guess you would call it performance testing, starting maybe like a few weeks ago up until November to make sure that we are ready just to give you some numbers on that right now at peak, we need to be able to send 83,000 emails a second and we need to be able to ingest about 150,000 events per second. Ultimately behind that is supporting somewhere between 1 and 3 million ops per second on our Cassandra cluster, which is our NoSQL data store that we use. So we know the numbers that we need to hit. And again, the site reliability engineering team, we're responsible for providing the automation and tooling to be able to help engineering teams spin up infrastructure that they can then run their tests on. But it is the responsibility of the product engineering teams to know what numbers they need to hit as well as to hit those numbers in load testing before the big day. I will say that we helped them come up with those numbers and we helped them come up with plans for meeting those numbers, as well as facilitating conversations between product engineering stakeholders to make sure that the numbers were correct. But at the end of the day, we more support. It's more of the carrot and less of the stick. We want to support them and guide them but not tell them what to do because ultimately they're the experts in their part of the application, they're the domain experts. We just support them.

How do you know what type of infrastructure to scale up though? Is it the same infrastructure you would use in production so that you know whatever you're testing and staging would also translate into same performance in production?

It's a really good question. It's a hard to answer question because number one, due to the age of Klaviyo and number two, due to the sheer amount of data that we have, it's not exactly possible for us to test in staging the way that things would perform in production.

We could probably invest in it, but it would be economically disadvantageous to have all of our data as well as complicated from just a data privacy perspective, to have all of our data in staging to be utilized for testing. So frankly, what we do is we do have a staging environment that in many ways is representative of production. But in some Ways is not. And then we also test in production when we need to. Which is the best way to know whether or not you're going to be able to do it in production is to test in production. But we do before doing that, do a bunch of testing in the sort of ephemeral low test environments that we can bring up and spin down as we'd like to.

So Laura, I also noticed, I think you're the organizer of Pyladies in Boston. How much Python do you use in your day to day job and what's the group like? Does it help people learn more about Python to help them with site reliability engineering or just programming in general?

Yeah, so I use a lot of Python in my day to day. Again, our application for the most part is written in Python. It's written in Django, so it's a big Django monolith. And then there's some other smaller services that are also written in Django. Then the majority of the infrastructure automation that we have, configuration management automation, is written in Python as well. Although we do have some bash. I write a lot of Python in terms of Pyladies. Pyladies is not specific to site reliability engineering, much to my sadness. I would love to to get more folks involved in site reliability engineering through the Python routes, but frankly, if you are interested in learning Python, if you are a woman or non binary individual, you can come and learn Python. And it's not just about site reliability engineering, it's about web development, testing, data science. We cover everything.

Laura, as we mentioned you have two courses currently on LinkedIn. One is on DevOps foundations microservices and the other one as we mentioned, graphite and Grafana. Any other courses you're working on? You seem very busy, but I'm just curious to know any other courses, any other subjects you think people should know more about and that you plan on putting out more information to help them with it?

Yeah, there's actually a course that I'm going to be recording this week about it's DevOps foundations and I think it's called getting into DevOps, something like that. And it's just going to be a course around, hey, I'm interested in maybe becoming a DevOps engineer or a site reliability engineer. How do I get started? And that's what the course is entirely about. So if that's something that you're interested in, please do be on the lookout for it on LinkedIn learning. I think it's going to be awesome.

Okay Laura, before we go, is there one piece of actual advice you can give someone to help them improve their site reliability efforts and let us know the best way to find or contact you.

I think the best thing I would tell someone to do is to stay curious. So when you encounter a problem instead of saying, oh well, you know, this is something that's outside of my current domain, you know, I don't really know that much about Linux and I'm going to let someone else solve this problem. Really dig into the problem and go all the way to the root cause. Dig all the way down. Dig really, really deeply until you know exactly what's happening on your computer. That is the piece of advice that I would give to folks. Stay curious and be patient. With consistent effort you will get it done and you will learn what you need to learn. So that's the piece of advice. And then in terms of getting in touch with me, I am happy to chat more with folks on Twitter. Tyrostone is my Twitter handle so would love if you wanted to chat. Feel free to message me. Feel free to follow me. And yeah, thanks so much for having me.

Thank you Laura for your site Reliability Engineering Awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com p likeperformancep02 and while you're there, make sure to click on that Try It Today link under the Exclusive Sponsor section to learn more about Smart Bears or awesome Load Ninja Performance Testing Solution. So that's it for this episode of the Test Guild Performance Podcast. I'm Joe and my mission is to help you succeed with creating automation Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.