AgileData #19 - Refactoring, magically making changes when you need to

Welcome to the Agile Data Podcast, where Shane and Nigel discuss the techniques they use to bring an agile way of working to the data world in a simply magical way.

Welcome to the Agile Data Podcast.

I'm Shane Gibson, and I'm Nigel Vining.

And today Nigel and I are going to have a bit of a chat about refactoring, something that we've been doing a little bit at the moment in Engur, and something that we, we have actually been doing a little bit over the wee while that we've been running. So for us, it's something that we, we plan to do something that we do regularly, and we kind of want to have a chat about what it is and why you'd want to do it. So, refactoring effectively is rewriting something that you wrote before or replacing something that you built before. So, the idea being that often when you build a feature or create a moving part within your architecture, You are often time-bound. So, you have a bunch of choices and you pick one of those choices knowing that actually if you wanted to spend a lot more time and a lot more money and a lot more effort, you probably would make a different choice. But with the context of the time you have, choice A is the one that fits you the best. And you do that on the basis that you know you have what we call technical debt. So, you know that you have some money that you have to pay back and you pay that back by refactoring. So give me an example. You might do a quick prototype where for some reason you're building an app and you decide that because it's only you and your development team that are going to use it, you'll just whack it out there with HTTP. So there's no security on that call because that's the quickest way of getting a web server up and running. And then you may, as you get closer to actually making a proper one, you need to replace that front-end web server a web service with an HTTPS one, with a secure one. So you'll go and swap it out. And ideally, the cost of refactoring will be low. You'll design it in a way where you can just swap those two components out. So for us, actually, we've done constant refactoring as we go, and we're doing a major piece right now. So the config which holds all the magic within Agile Data is actually right now stored within BigQuery in Google. And that was a decision. That we made.

Yeah, that's a great intro, Shane. I was gonna say, I always know when it's time to refactor when it becomes hard to change something, and when your product owner, or in this case, business partner, asks you to do something, and you start to grimace and say, "Yep," and you're suddenly not so sure. What started out as a 5-minute feature in the first, in the early days, suddenly becomes half a day of tweaking to get it work, that's usually the sign that it's time to refactor because you've outgrown your initial pattern. As Shane said, a lot of the config magic for agile data I/O, we started out using BigQuery because that's where our data was, has heaps of functionality, and it was really good at the time. It allowed us to really quickly prototype, iterate, and get the first MVP out to the initial customers. And that stood us in good stead for literally 12 months now. And then recently, our last customer we onboarded, Shane, started to ask for additional features that didn't quite work. And we've had to effectively bite the bullet and refactor a lot of that under the COVID stuff and take all our learnings, take the new stuff that we want, and build another version. And that's what we're doing right now.

Yeah, so for us, it was something we knew we had to do. We planned that the amount of time that the current pattern of storing that config in BigQuery would have a natural end of life. We had some ideas around how we would refactor it, which of the other Google services we would use under the covers to do that. We've actually— I had a guess a while ago which one we'd use, and we've actually changed our mind on that for a whole bunch of reasons and found something that's a little bit better and a little bit more cost-effective and a little bit faster. But we knew we were going to do it. So, there was always going to be a logical time when that refactoring needs to happen. The benefit of doing that is that as we build something, we build in a way that we know we're going to have to rewrite it. So, we don't overinvest in things because that gives us more effort next time we refactor it. So, we don't add superfluous worthless features in there because we have to write them again. And no doubt, you know, once we finish refactoring where we store this config, you know, in the future we're going to find some reason that we'll have to refactor parts of it again. So we plan for that. So we only add features into it when those features have value, when we really need them.

That's, yeah, that's a really good point. Sorry to interrupt there. We've already, our last walkthrough of the new the new pattern, we've already started to say, when we change to X, when we change to Y down the track, because there's already additional refactoring we know we're going to do in the future with additional features from the app on top of this config. So we've built it in mind that parts of it are going to be replaced again. So they are like an interim placeholder for now. They'll do the job until we deliver the next piece. But at that point, they'll get substitute it out again. So we haven't— they're probably not as robust. I haven't invested days and days. I've done the minimum so we can use it till we transition it again.

And I think what helps us be successful when we do this is everything we build is— I call it Pac-Man, but it's a service. Um, so the way I describe it is every component we have is a little Pac-Man, and Pac-Man eats something and then Pac-Man deposits something. And so if we think about that, we're able to take one of those Pac-Man and replace it with a different piece of technology, or we can take that one Pac-Man and break it out into three Pac-Man, right, that takes it, does something to it, passes it off to the next one. And we may go through three new services to replace that one service to give us more flexibility. But again, thinking in that that way of working where something comes in, some stuff's done to it, something comes out, means that the impact of refactoring each component is known. We know what the blast radius is, we know how many services are effectively affected, and therefore, we can have a guess at how much work's involved. So, you know, when we refactor a rule to add, say, you know, one more way that that rule could work, so give you an example, you know, you might have a rule that changes the case from uppercase to lowercase, or lowercase to uppercase, or, you know, into camelCase. Those are small changes to the rule versus the one we're doing right now, which is refactoring one of the major components, one of the major services that we rely on. So we know that a slight rule change takes a short amount of time. We know that this level of refactoring we're doing right now means that Nigel has to stop and focus on that for a longer period of time to get that one done and make it safe.

Yeah, yeah, it's a, it's a big investment of time, but it's, um, it's also really useful because during over the last X number of months we've built up a backlog of additional features that we keep saying it'd be nice if we could do this, we could— nice we could do that. And this is a logical point to grab a whole lot of those features out of the backlog and insert them in because we've effectively unpacked this pattern, we're putting it back down again. It's very little effort to tweak it slightly to give it more functionality as we do that. So, you know, refactoring is also useful time to bring in enhanced features if they've been sitting there waiting for that opportunity.

I think one of the other things that's interesting is because we are refactoring we kind of break it down into two chunks of work. So as Nigel said, we have a bunch of features or, you know, in this case, rule types that we want to add that we were struggling to add to the way we had designed that initial piece. But what we do is we logically work into two iterations effectively. One iteration is to refactor the current state with the new technology pieces. And then stop, because then what we're able to do is run systems in parallel, confirm that by refactoring, we haven't broken anything, we haven't changed any of the behavior, all our tests execute. And then once we get to that non-state, then we can do the second iteration on that, which is starting to add in those features that we had backlogged. But by having those features already known, while Nigel's not working on them as part of the initial refactoring, they're in his head. So, as he's refactoring, the Pac-Man or that service, he's thinking, oh, well, actually, that feature would be easier to do in a minute because of the way we've done it. So he's kind of thinking ahead of how he's going to add those new features in as he's doing the refactoring, but still focused on refactor it, make sure it still works, gives us the same results that we had last time, just happens to be more flexible and faster and ideally costless.

Yes, what just came to mind, actually, while you were talking was my other role that I'm doing at the moment on the side. We're also going through a piece of refactoring, but it's, I guess, it's in a context of a company, and we're refactoring underneath code that the customers never see. So, it's been quite a more difficult sell for that investment of time because the end customer actually doesn't see any change. They still get their reports, everything still works, the covers, but the refactoring is effectively to bring efficiencies and reduce technical debt under the covers. I think in our context with Shane, it was a pretty easy discussion saying, we need to refactor this because it'll make it faster and easier in the future. Whereas generally refactoring's potentially not as positive because the benefits not as readily available. You don't see any change, everything still works the same. The fact it's running quicker under the covers, it's not as easy to sell that one.

Yeah, I think because we've been working in an agile way for so long before we started agiledata.io, We knew that technical debt would get built up no matter what you do. We knew that refactoring was a way of life. And so, really, the conversation is not about if we're going to do it, it's about when. But, you know, in my side hustle, when I'm out coaching teams, I have exactly the same problem as what your customer does, and that the stakeholders, they don't see the value of refactoring because as far as they're concerned, paid for some time, they got what they wanted. It's not their problem if it runs slow or if it's fragile or if you can't make changes to it. So, what I always say to the teams I'm coaching is, you know, if you're working in a Scrum or an iteration kind of framework or way of working, that they need to book the right to have a technical debt iteration and refactor. Or, so, sorry, in that scenario, You know, you might be doing 3-week iterations. You might say every 4th or 5th iteration is refactoring some of the work and paying the technical debt down, and there is no new information product being delivered. There is no product owner with a bunch of stakeholders expecting additional value. So that's one approach that a team will often adopt. The second one is what we call points holdout. So the team has a certain velocity, they have a certain amount of work they can deliver every iteration. They reserve the right to take a percentage of those points and use it for refactoring previous work. So the interesting one about that is, is effectively the product owner who's, who's in the current iteration, who's getting the value for their stakeholders, is paying a tax for all the previous product owners. But that's okay because if you explain to them that actually if the team's automating their work as they should be, the previous product owners have worn the tax of initial build of services and features that the latest product owner is getting. So they're just paying it back into the co-op, into the pool. So yeah, I think if you're working in an agile data way, you need to make sure that you have the right and the ability to pay down that technical debt, to refactor what you're doing. Because if you don't, you're building a house of cards, and it will fall.

Yeah, I love that analogy, paying some tax on the features that you've already been delivered. That's a really good way of looking at it, actually. When you phrase it like that, you can't help but get by, and everyone has to pay a little bit of tax for that debt, accumulates and gets paid down as you go along.

I, I have a Nirvana one that one day I'm going to work with, uh, a customer that, um, actually records technical debt as a debt on the balance sheet, uh, and there's interest, or they record it as an asset that gets depreciated, uh, and so therefore there's depreciation. Um, so in fact, actually, it's something we should probably think about is, uh, you know, how would we put some form of data against the refactoring we know is coming up? How would we do very light estimation to understand when we have to pay it down?

Yeah, no, I like that. I like the idea. I don't know how you quantify it in advance, but I think it, yeah, as a placeholder, you know it's always going to come around. Yeah.

So, one of the things that, you know, often a new company will do when they bring on somebody new into the organization is they will put that person on the front line. They'll put them into the support team so that they can get an idea of their customers and what the customers are doing and how the product works. From a refactoring point of view, what would you reckon would happen if every time you brought in a new developer, you gave them something to refactor. Do you think that would be a good idea, a bad idea, or a dangerous idea?

That's an interesting question. On one hand, they see the problem or opportunity with fresh eyes that aren't tainted by any historical cultural knowledge or people have come before and tried that problem and put it aside. So I guess they come at it with a fresh set of questions. Why are you doing this? What's the outcome of this?

Why?

Lots of whys and whats. And then I guess the flip side is directly is they're approaching it with no prior knowledge of how it got to that state. It's a tricky one. I'm not sure I have a— I'd be on the fence about that, wondering if it's good or bad.

Yeah, I mean, some of the stuff I've read and case studies I've seen, we talk about bringing in a developer and within the first week of them starting, they push something to prod. They release a feature that hits the customers. And the idea being there, they see the end-to-end process. Now, obviously, it's got to be something relatively small, you're focusing more on the process so they know how to develop and how to push versus building a big, massive feature. But yeah, for me, that was the other option, right? As you find a small piece of refactoring that needs to be done and you get them to do that in the first week. So they can pick up and see how each of the bits are built, refactor it in some small way, and push it. Get that sense of achievement.

Yeah, I guess that's, um, now the analogy's sunk in a bit more. Um, recently we brought on a new developer onto our team, and the first thing we effectively did was gave him some code that had been written by someone else, um, and we gave him a few days to go away and basically make an assessment. Was he gonna take that code and run with it? Was he going to rewrite that code, or, you know, what was he going to do with it? Which was an interesting example because he went away and he thought about it. He did obviously had a play with it and got it to work, and then he came back and he's taken on a bit of a hybrid approach. He's taken half of the previous developer's work, uh, he's kept a portion of it, then he's basically brought his own flavor to it as well and he sort of started again, but it was quite an interesting experiment to see which way he would go with that.

Yeah, actually, I didn't think about that one. I was like, maybe that's actually something you can do before you hire somebody. So, we all know people that have their standard modus operandi is to go in and go, that's all crap, I've got to rewrite it all. And, you know, yes, sometimes it's easier to rewrite something, but if that's a way that that person works consistently, then it's a little bit dangerous. So, yeah, maybe that's a good way of having a very short interview process is provide some examples and say, how would you deal with this?

Yeah, I think it was quite relevant because, A, it showed that he, A, understood what had been written, so he understood the language and the patterns in play. And B, he was experienced enough to see the weaknesses in the existing pattern and suggestions of how to improve it, which, you know, I thought was quite a valid test per se.

Yeah. And often, you know, code's written with context. It's written with the context of what you're trying to achieve. It's written with the context of your experience with whatever you're working with, and it's written with the context of the time you have to get that thing out. So, you get long enough, you write beautiful code, but often, you know, you ship a good enough feature, and you know that you can refactor it. So, for me, I think that's really the key point is nothing sacred. Everything you write, or everything we do in terms of the platform, we know we'll probably go back and touch it again. We will definitely go back and make it better. And so, that's just the way we work. And then, the key thing is making refactoring safe, right? How do we know we can go back and touch a core service, a core Pac-Man that everything relies on, and it won't bring the whole thing down in a house of cards? So, that ability to understand those moving parts, those services, and how they interact, and how we can make sure we know when they're behaving the right way is critical. So, for me, it's kind of plan to refactor and then make sure you do. Otherwise, you stay static and that's probably the worst thing you can do.

Yeah, I think that probably sums it up nicely for me as well. We're going to refactor everything again a number of times. It's just how you grow. You will outgrow these patterns. Maybe they won't refactor much. Maybe it's just a little bit of finessing around the rough edges, but, you know, 80% of the current pattern may live on for the next couple of years quite happily, and we may just tinker with some of them, but we will tinker with it because we will outgrow it eventually.

Yeah, yeah, if we don't, we don't outgrow it, means, uh, we've gifted at guessing or we're just not pushing hard enough to build cool shit that's simply magical. So, all right, I think we'll lock that one out and, uh, we'll catch you all next time.

And that, data magicians, was another Agile Data Podcast from Nigel and Shane. If you want to learn more about how you can apply agile ways of working to your data, head over to agiledata.io.