AgileData #13 - A DataOps and Data Science Journey

Welcome to the Agile BI Podcast, where we chat with guests or sometimes just to ourselves about being agile with teams who are delivering data analytics and visualizations. Welcome to the Agile BI Podcast. I'm Shane Gibson. I'm Hamish. And I'm Liam. Welcome, chaps. How's it going? Well, we're recording from a— not from Wellington today, and we're recording in a pub, so that's a first for me for both of those things. So I think we'll probably start off with Hamish, give us a bit of background about your journey into this world of agile.

So I guess my story in data analytics starts back in probably 2000 in the UK where I fell into data warehousing through Access databases and then into a few BI reporting tools. Lived in Australia for 7 years and then back to New Zealand 5, 6 years ago, and the Agile journey started probably in January when we met Shane who came in coaching our team where we work.

Liam? Cool.

I've only been in the workplace for about nearly 2 years now, obviously straight out of university. My journey in data analytics— I'm obviously a data scientist. My progression in the workplace has started around a team which traditionally did reporting to a team which now does predicting. And then also there is some of the buzzwords around, like AI and machine learning and predicting and models and all sorts. So I kind of came in to help try and connect the dots on some of those things. And try and be a team which is more—

try and do more things.

Yeah, it is.

Better things, more interesting things than just reporting.

Yeah.

I think before we met Shane, we were stuck in a BAU support model and we got the opportunity to create a new platform, cloud-based platform, and with that came an opportunity to do things differently and work better as well. And that's been probably in Liam's job has been on more of the spiky end of that with all the maths and things like that rather than the crunching of rocks, which is probably more my data engineering department.

And I guess we've now gone down this agile route for about nearly a year now since our first training to then running a couple of sprints. So it's been actually quite an interesting journey and we've learned kuaolong. Quite a lot along the way.

So, I think today we're gonna talk about that journey, and then, kind of, like you said, it's been— hell, has it been almost a year?

Jesus.

Although we kind of— I think we went in there slightly lightly to begin with before we kind of doubled down and—

Should we go kanban? Should we go sprint wall? I don't know the difference between those. We'll work it out as we go.

Yeah, and that's interesting. I suppose, you know, I was just telling the people on the course today Often when I work with a new team, they want to go Kanban because it feels comfortable. It's kind of like the chaos that we normally run. And in the past, I used to always say, no, you have to go Scrum because I'm more comfortable with it. It's more of a— we put unnatural constraints in. It helps you kind of learn some of their job behaviors faster. But as with your team, you know, it was like, well, yeah, give it a go. Let's give it a go and see what happens. And then if it doesn't work, if it doesn't seem to work for us, we can bring some more of the Scrum behaviors in the beginning until we kind of learn those and move to the Kanban flow-based stuff. So, I mean, how'd you guys find that? How'd you find trying to use a flow-based model on day one before we kind of flipped and became more constraints-based?

I think you're right, it was fairly comfortable. I think everyone's worked in that way before, and even in other teams we've had tickets, maybe not deliberately kanban, but on a wall that you'd move between those statuses. So that worked fairly well to start. The thing that helped the team the most was the renew— was the focus we got out of it and the things around being able to ignore other people a bit more. You gave us permission to focus on ourselves, which we did fairly well to start, and that had Interesting side effects as well, I think, which wasn't always— didn't realise that it would cause— other people would look inwards at us and wonder what we were doing, and we were just trying to get on with what we were doing. That was our assumption, at least. So, karma at the start definitely helped, but it wasn't— we knew it wasn't gonna last forever, I think.

Yeah.

It also took us a couple of attempts. We thought, 'Ah, yeah, we might be able to implement this relatively quickly,' but then you kinda said it might take a wee while, And then we kind of thought, "Oh, we should be able to have a product owner that doesn't need to give us 50% of the time." Then we kind of went along that journey and figured out pretty quickly that—

I mean, we're still just having trouble with product owners now. We're not— we've had good, we've had—

But we understand, like—

And they've got probably the hardest job, and that's what I tell them.

Yeah, no, I agree. I think it's one of the hardest things I find to get good, dedicated, committed product owners. That are there, right, for the team. I think it comes back to that waterfall behavior that, you know, you can throw a set of requirements over a fence and, you know, wait and then go back right at UAT time to tell the team that it's not what you wanted versus spending that time to engage.

Yeah.

You went through quite an interesting journey where you started off with quite a strong product owner and then in a couple of the sprints you got one that was a little bit less available, and then back to quite a strong product owner again in terms of the next person. So how did you find that in terms of that seesawing between somebody who was committed and dedicated to then kind of involved and then back to another one that was committed and dedicated again?

I think it was knowing what decisions to ask them to make, or we'd often— we got into the habit of we'd start making decisions for ourselves. And then it became a motto, "Let's ask the product owner." And when we had a product owner who was there, as in, whatever, 2 ago, 1 ago, then we actually would start to engage with them constantly, and it did work. Whereas if they're absent, one of us would just take the lead and just decide. So which has been our probably worst effort, I think, and it is down to the product owner giving us direction, but it's It depends how the information products come to us. Some are forced from up on high and some are generated from an actual need. And having the person understanding what it is that needs to happen versus just thinking this is something that someone's told them to do, then doesn't always work. So, but definitely with the Engage Product Owner is, was the best we've had, but unfortunately that priority's changed and we're not on that at the minute and then we're on something else. That's kind of another story.

Product owner thing is something that we've tried because I guess if they don't really know what the role of a product owner is as well, we've kind of had to then coach them a little bit on that as well as then engage them in a sprint scenario as well. And I guess also the different thing is that we don't have the luxury of being able to, I guess, have that as a prerequisite, as in, okay, this product owner does roughly know what the product owner does before we actually commence the sprint. Usually we'll be then commence the sprint and then kind of teach them, okay, can you please do this and just make sure you keep an eye on the tickets and done and things like that. So I guess we're slowly sort of refining the product owner onboarding as we kind of get further down. And I guess it was only kind of one way to learn that, and that was through doing, which is what we've heard multiple times.

Yeah, I mean, I think One of the things I find interesting is that if we're looking at, you know, what we call the traditional agile approach where we're doing software development, we will typically have a product owner for the lifecycle of that product. But when we do data and analytics, because we deliver something to a certain part of the business or for a certain business process or a certain outcome, we seem to switch product owners in and out more often. So we don't have a dedicated product owner that takes us through that journey. We have a product owner come in for that outcome and they kind of exit and then come back in later. So I think that makes it difficult because we are recycling them. And I think your point, Liam, yeah, yeah, there's a— I've started to realize there's this role called a product owner coach, which is, you know, helping those product owners on board early. So, you know, from a definition of ready, we try and groom the tasks before they hit this, the next iteration, we should probably, and I'm not sure grooming a product owner is the right term you can use, but we should be helping that product owner come up to speed before they, you know, the rubber hits the road. But then if we're struggling to get their investment for the length of the iteration where we're doing work for them, getting that effort early when actually they're just learning how to do that role is going to be challenging as well.

Yeah, and I think the successful product owner we did have we spoke to their manager and made sure that they were aware they're coming down to sit with us 50% of the time, and we actually did that bit well, whereas on other ones, not so much, I think. So that was probably the difference we made, and then we were very clear, by the way, your role is to make sure you talk to the stakeholders. I'm not gonna talk to them and make sure they're aware of what we're doing. You need to keep them all on board, and they knew that's what they were doing, and actually, A problem we had a while back was we had stakeholders come to the demo day, and then they're like, "Well, what use is this?" And we assumed that the product owner had been talking to one of the stakeholders, "It's no good if you do it every day. It needs to be done every— more often than that." And we're like, "Okay, we could do that." But the product owner never knew that their role was to talk to stakeholders and make sure that their requirements were met too. Which comes back to product owner training, I think.

Yeah, and I think that's critical, especially as we recycle them, or cycle them in and out, not recycle them as in put them in the crusher and bring them back again. Yeah.

Now, we do have a little bit of documentation around, because everyone uses the same sort of terms like done, in progress, and so forth, and we had to then make our own little documentation of our own definitions, I guess, our little flavor of it, because we learned quite quickly that, yeah, the Agile definitions in some different parts were quite different. So we just had to make sure that we did write that down on a bit of documentation and then show that to them. So that's kind of part of our onboarding is having that.

Yeah, I mean, that's a funny one, was with another team and They have a stakeholder committee, a governance group, which is something that they can't not do at the moment. So, you know, agile delivery of the team, but not so much business agility. And so they were having a steering committee, and the steering committee had actually had some agile training. So the team were presenting the information products and what was needed in terms of their priority, and the stakeholders said, well, you know, what's that? Is that an epic, a feature, a user story? And we hadn't quite hardened our terminology up, so, you know, you can have a bunch of epics within an information product if you want to, you know, you can have a bunch of features within, you know, and so as part of that, kind of hardened it up to say, well, an epic is an information product. You may not want to have it that one-to-one, but actually by having it one-to-one, it's really, really clear. So having that terminology is really important, I think.

Yeah, we quickly found that we ended up having epics and tasks, and then everything else in between was just— we didn't think about. We didn't think about features. It was more acceptance criteria, and we kept it down to that. So we've only really got two things: an epic is an information product, and and then there's just the tickets we need to do to get the thing done. And that was what we did. Having the rest of it, we found just too much effort for probably the size of team we got. Small team, we didn't need to have all these different things. But success criteria, having them, and not lots of user stories, but enough to make it usable, but probably the, what does success look like? That's worked well, hasn't it? Having those things, and then we can work to them. Probably don't refer to them as much as what we should, but having them set up front allows the product owner to see what they're going to get.

That idea of acceptance criteria, how are you finding setting those? Because I tend to struggle, right, getting good acceptance criteria, especially with data and analytics.

Exhausting. Yeah, there was the one we did with the good product owner and you weren't there, Shane, and Liam and I pretty much had to run the By the end of the planning session, by the end of the day, I was knackered because we had to draw out of them all the things we needed to know and it was just— ended up getting a little bit technical, but we've got things and we can point to them and say that's done and not done.

But it was—

it's hard work and it takes longer than what you think. The planning, probably the big thing we learned is the planning and the retros and all those type of sit-arounds. Just take ages and it's at the start so much time spent doing it, but it's useful. You spend a day and a half planning, then 3 days executing it, actually it's worthwhile rather than just the opposite of that, which is probably how it used to be. Yeah.

Yeah, I mean, interesting question I either got asked or asked somebody the other day is, you know, of those 5 ceremonies that we do for Scrum, which one would you lose? If you had to lose one, which one wouldn't you do?

We hardly do backlog refinement, actually, to be honest with you. Backlog grooming, we rarely do that just for the—

Is it because we've got someone remote or— Nah, it's not really because of that, it's just—

The retros we all like.

Yeah.

Gives us a good time to—

Yeah.

Download.

Yep.

Have a moan.

Do you think that not doing the refinement as often as we normally would is a symptom of not having a really good IP backlog? That there's not really a bunch of IPs that have been prioritized and that you know kind of which one might hit you next. It seems to be slightly more random in terms of which priority's the priority next time, which, you know, is okay, right? From an agile point of view, being told just in time what you're working on is okay. It's not great. You kind of want to have a bit more of a roadmap ability to kind of groom and think about and refine the stories before you hit planning.

I think with us there was a wider company agile thing kicking off at the same time, and I'd go to bigger planning sessions and they say— well, I'd say, we can work on that in 2 weeks. They go, how can you work on that in 2 weeks? Because our current sprint ends in 2 weeks, and everyone was just, oh, you can't say that. It's like, well, we can, we'll work on whatever's big next. But you're right, beyond whatever the current sprint is, we never have a view of what's next, or we haven't because it's, what, changed 3 times last week, so it's not ideal, but that's what's been happening to us lately.

Yeah, and we did a bit of portfolio planning, I remember, right? We tried, I think we had 2 or 3 goes where we tried to get a roadmap of what next would look like, and then as soon as we walked into the next iteration, priorities had changed, and so You know, we kind of decided that it wasn't worth the effort. We were putting a whole lot of effort in for something that was giving us minimal value at the time, for a whole raft of reasons.

And I think our transition from a reporting team to a pure analytics team has been harder and longer than what we would have hoped due to other factors that we can't control. So we're stuck with a lot of BAU things that are not pure analytics, but we get stuck with them and that just seems to be stuck with the people who've been there the longest and there's these types of problems we get to. So we've, we always talk about BAU bleed and we've never really got a good solution for it. So I think we're just learning to accept it now and just, just do it. Timebox it doesn't always work, but that's some of the approaches we've been taking.

Yeah, I used to think around acceptance criteria because when I did a little bit of Scrum Mastering for just a small DFTM, what, 2, 3 people, and the acceptance criteria, I was kind of alliterating that that was like the lifeblood and that we didn't usually, I think in one of the sprints we didn't quite prioritize, we just had the acceptance criteria in bullet points, or we had like things as like high priority and would give like 3 or 4 of these tasks a high priority, but then we actually looked at it, okay, which one of these higher priority has a higher priority, so we've kind of over time managed to kind of kind of also, I guess, prioritize that acceptance criteria. And I guess the next step also is, I don't know if something we look into is whether or not we link some of the tickets to the acceptance criteria or something, because we don't actually have a direct link. We sort of do, but there isn't necessarily anything showing.

So your acceptance criteria are kind of more sprint goals, right? Yeah. What success for the sprint looks like. And then as a team, you're mature enough to decompose compose those goals down to a series of tasks. But I think I'll come back to your point about we're not really doing user stories or features, and that's probably because, you know, we're walking into the iteration for sprint planning with a product owner and a goal we haven't refined. You know, we're going straight to, well, what's the acceptance criteria for the sprint? And then how do we apply tasks to get there? And the team's mature enough that it works.

Well, in some cases though, we end up with a lot of off-task, off-ticket tasks. Hamish is always working on that off-ticket task, which is the combination of the 5 other tickets. So what we often find is the task creation comes at the end of the day, everyone's tired, so we just smash them out. And then we've stopped doing that, and actually the next day, it's like, we'll do them all again in the morning when we're all fresh. So it ends up taking a day and a half to do the planning, and then the tickets are getting more task-related. But because we're working in new technologies and we're learning constantly, it ends up being a, you know, we've got more than one way to do things. Sometimes we, today even, we changed our mind on how we're going to do it. And then, so what ticket are you working on? Oh, it's actually all of those 5 all at the same time. How can that be? It just kind of is, and then they all move. So that's where backlog grooming would help us, where we actually— do we add tickets during the sprint, but it ends up being a ticket because it is a thing. So it means our estimation and velocity is always out the door, really. It's—

The story points to tickets and the whole burndown chart that we've been looking at as well, like it's kind of— The story point thing, we're still really trying, working hard to try to get sort of, I guess, correctly sized stuff, right?

But what would take us 20 points in January now is seriously 3 to 5 points. So we've evolved our platform to the point where we can do stuff quickly, but then there's always a new— cloud-based stuff, there's always heaps of cool little updates they do, and you just go, Oh, shiny! I'll just upgrade that now. Oh gee, maybe I shouldn't upgrade a terraform. I'll roll all that back, but that was a whole morning wasted on trying to. So we've got little things like that that catch us out all the time, and I like shiny things, so try to avoid doing upgrades mid-sprint. But it doesn't always work. No.

But again, you know, you've been building the aeroplane while flying it, right? Yeah. How have you found that emerging architecture, that emerging design? That actually building the platform as you deliver value to the business users versus what we would call iteration zero, where you're given 3 to 6 months to build something that met everybody's needs. If you had to go back, which way would you go?

I think the way we did it, to be honest with you. It's been— we don't know if we talk today, someone asks, well, why'd you do it that way? We're not going to build anything we don't need to use. In advance. We can't— we shouldn't assume we're going to do it that way. So we've built some things that we have changed, but we've built a lot of things that we thought we might have to change that we've never changed. They've ended up being the right decision. So it's better to make a decision than try and figure out which method was better. We've got 3 ways to do one thing, but we'll slowly migrate to one, or we won't, or as and when we need it. So, I mean, a good example of that is our onboarding documentation, which is non-existent. So new people find it hard, but we're never had to onboard anyone until last week. So now we've got people, now we'll build it, we'll do just in time. And I think, I don't know, it works for me, it makes it fun, makes it interesting. So you've got to think on your feet.

But how much refactoring do you think you had to do over the last week?

Well, we haven't refactored. We did take actually about a week and a half of refactoring at one point, which was a good decision. But other than other than that, there's not been, there's now two methods for doing the same thing, which we need to merge into one, so, but the idea of being able to build everything, burn it down and build it all up from the beginning is still the idea and it's, we can't do it tomorrow, but that's where we want to get to, where if we had to refactor it all and change all the metadata columns for someone had a different, someone who used camel case or we wanted to change change our naming conventions, we could do it and it all should just work, but it's not there yet.

I guess using your aeroplane example, it's kind of building it while it's in air, but I guess as kind of a user of it, I guess I see it as sometimes, you know, when you go to land, you're kind of quickly refuelling half tank and then quickly bang up in the air again and onto another new sprint. So it's whether or not that kind of the trade-off between up in the air or whether or not you stick down, I guess 'cause we've been doing sprints back to back to back, right? I guess we've only really had—

We're either lucky enough or smart enough that it hasn't crashed yet either, so—

But is that— Yeah, that analogy is not so much that as you're building the aeroplane, it's not good enough. I think the analogy you're using is that we call it a sprint, but actually it's a marathon, right? Where we sprint and then sprint and then sprint and we never actually stop. So there is no respite, right, from delivery of value to the stakeholders. There's no gap. And I was reading something the other day or a podcast or something where they were actually saying that that constant work, even when we get good at making it just, you know, 40 hours a week or whatever standard week is, still burns us out. That actually we need to inject downtime. So they often, you know, they were talking about leaving every couple of sprints, taking a week out where you don't actually do anything. Yeah, you always stuff to do, but you're not actually in the sprint cycle. So that's an interesting thing.

Yeah, we could do with that right now.

Yeah, the one I I tend to, I like using hackathons, right, or innovation sprints where you're still doing something, but it's more fun. So yeah, one of the teams I work with, they do the Dragon's Den. So they were a lot bigger. So they actually, Gerhard was on here talking about it a while ago. So they got the whole team, I think they had 20 or 30 people, and they split them up into teams of 5, and then each team got to pick something that they wanted to build, and then they all just spent a week building it, and then they presented to the senior executives, the Dragon's Den, to get funding to be able to then put it into an iteration and deliver it into production. So that gave them, you know, they were still doing work-ish, but you know, that gave them a bit of something different, right? Yeah, a little bit of variability. Or all that seemed to make a difference for them. And I think that's one of the key things that, you know, need to recognize for you guys is that your team's tiny, right? So, you know, how many people have you got on your squad for this on average, you know, because it's been a slight change?

4.

Yeah. So that's, you know, down to the minimum viable team size, right? You know, very, very small team for building an entire platform and delivering value.

Yeah, I think in some ways we enjoy that because then everyone's always got something to do. There's never a dull moment and it makes it interesting. Like, some of our deployments have been pretty good fun even though we're doing them on Thursday when Demo Day was on Friday, even though that's it. But it was lots of high fives when it finally went in. Why should we be surprised? Surprised it works.

So yeah, that's right.

One of the things we put in at the start is having tests for things. So we've got pretty good at getting, getting tests in, and then we do slack every now and then. We've had some stuff lately which is kind of a classic ETL process where we haven't got a test for the actual result, but we need to go and put in tests to make sure that when we've got shared components and we change them, that that ETL we built— call it ETL— 2 months still does what it says, what it should do. We want to change these things. We've got— we're slowly building up a little bit of, not debt, but features we need to add to make sure that everything keeps running smoothly because we've got 2, 3 things in production now and one of them is publicly facing and one of them's, you know, getting consumed daily by a team of 10, 15 people. So we're building that critical mass now where we just can't destroy it.

Can't hack it.

Whereas before we could, we could afford to get away with it. Now we can't.

I think, you know, you have a very strong testing mentality and you've also automated the testing a lot more than I normally see. Why do you think that is? Most data and analytics teams I work with, they struggle with automating testing.

I think we set out to do it and we've been lucky enough to work with a few people who've got a good background in different testing technologies and I think I've worked in warehousing long enough to know that not having tests kills you. And the good projects I have done is when we did have ways to burn it all down and stand it all up and build tests. And now they're all code-based and we're using different frameworks. And I think that's just experience. You don't want to be having to get up at 6 in the morning to check that something's completed. You want it to just to work. And so that's having done that lots of times, that won't do that anymore, and that's probably where it comes from.

But, and also being that small team, right, you can't afford it for one person to be the, uh, the 2 AM wake up and fix it person, because then, you know, they're not, not top of, uh, top of form for—

and I think we've built redundancy into things too, whereas some of the things we've done can go stale for 3, 4 days before it'll actually make a difference. So we've thought about it in advance, we've got that, give ourselves time to fix It's not daily loading, it's whatever, it's got time, it's not streaming data, which we're lucky about, but if we did, we'd up the level of testing to make sure we can handle that stuff.

So it's typically unusual still to have an engineering team that's building a platform with the science team. Normally it's, you know, the engineering team go build something and the scientists either build something on the side or they wait and do it. How do you find that being part of the same team, you're trying to build data and a platform at the same time where you're trying to use some analytical capability against it to see what the data can do for you?

Yeah, that's ongoing. Yeah, that's been hard. Before, before we start a sprint, we kind of give what sort of times we have available. So I guess when I do some sprint work, it may not necessarily be data science related, it could just be visualization sort of assistance. I guess I also— anything in data science is kind of hard to productionize as well. It's usually kind of a lot of one-off ad hoc sort of stuff. So yeah, being able to try and do any of that in kind of a sprint-like fashion. And we're still working through, I don't know if data science around agile and that work that good. I don't know, I haven't been around long enough. But the data science, and yeah, it's definitely been a bit of a road to go along. And there is kind of around the kind of machine learning stuff that we're starting to dabble with as well. The use cases for them aren't sort of at the highest priority, I guess, at the full team, because I've only got one team working. It is down the line, it is working on, but I guess we work on that sort of stuff outside of the traditional team sprint that we do, that sort of kind of outside of the sprint that we do.

So you kind of, I kind of play it back, you're involved in the sprint.

Yes.

And sometimes when you have capacity, you're helping out.

Yeah.

But by being involved in the sprints, you're understanding the data that's getting landed, what it's been used for, how much you can trust it. And then as you look, as you find analytical problems that you may want to work on, at least you have an understanding of where the data is and what's available.

Yes.

Versus being an intricate part of the team that are actually delivering a model in 3 weeks. At the same time the platform's being built, the data's being landed, quality issues being solved, and the business is being delivered, right? So, which would have been a bit of a stretch, I would have thought.

Yeah, and learning how to get the data and the transformations that go along with that has been, yeah, that value has been huge. So, yeah, learning some of the techniques of some of the other data engineers and that has been, yeah.

Do you think by being involved it stopped you going off and you were building your own analytical platform while you waited?

Yes.

Yep. Which I see a lot of, right, is that, you know, starts off as an analytics project because AI is the new cool, and then everybody goes, well, that's great, but actually we don't have any scalable infrastructure and we don't have any data. So, you know, I have seen customers where they hire data scientists, the data scientist turns up and they're like, well, seriously, you've hired me and there's nothing for me to run code on and there's no data for me to use. Yeah. So you hired me because— so, you know, that's one scenario I've seen that hasn't worked so well. But the other one is, you know, we just do data and traditional reporting and we never explore the analytical side. Yeah.

So, yeah, I guess also learning kind of the team culture and team behavior, being able to— because we we'll go out and we'll go and test some things that are probably in beta in some of these cloud analytics platforms. So the way we kind of learn how to approach the team members in the way that suits them, so we're not exactly hassling them so much, but kind of like, hey, look, we're keen to try to do this sort of thing, and how would you, you know, logging in a way which, you know, is easy for them and easy for us. So I think that sort of has been a great benefit as well, if I kind of if that makes sense.

Yeah.

Yeah, no, it does. I think one of the other things I saw that I found quite interesting was, you know, typically from a data point of view, we're taught to automate everything, automate our collection, automate our landing, you know, our ingestion should all be perfectly automated and bulletproof. But you kind of took a, you know, a file drop, I call it. I know you have other unique names for it, but a way of grabbing some data and landing it safely on the platform so you can then use it early. From an analytics point of view, that seemed to allow you to do some other things early because when you needed a piece of data, you know, you could basically go collect it yourself, drop it onto the platform in a more repeatable way, and then explore it versus having to wait for somebody else to do it for you or ask permission or, you know, log a ticket and wait for the next sprint. Is that approach of having kind of a robust way of acquiring more semi-manual, semi-structured data being valuable?

Yes, to be honest, we are. The thing that kind of Where the Cloud Analytics Platform has kind of come in was kind of the third-party apps that come with it and the ability to process large amounts of data. To be perfectly honest, some of the stuff that we've been doing, well, particularly on the data science stuff, hasn't been particularly too complicated as such yet. So the Cloud Analytics Platform, although it's been good, the whole robust thing, like, I wouldn't say we wouldn't have necessarily needed it, but we have big datasets coming up shortly especially sensitive ones, which will be perfect to then give it the full test.

But we did end up building two ingestion platforms, one that put the file drop, SQL Server, whatever else into the prod, but we've also built a— you haven't seen it, Shane— but the mini experimental zone uploader, which can do Excel files, all sorts of stuff straight to our database, which these guys can use. And I think the The trick is for them just using it by default, not slipping back into how am I going to do this, just load the thing and that's all. It was the whole idea was build a zone that they could experiment and do things and then lift and shift the code. It's not going to be a lift and shift, it never was, but it's almost we know your source, we know your preferred target, then we can push it into this. But without the UKR, we'd be in— it makes life pretty simple for getting new data sources, even small ones. Gets big ones, it gets, you know, hundreds of thousands of files, it doesn't really matter. So it's not—

and I, and I think that you showed us in the Expeditions Experimental Zone, and anything that we build, any tables get demolished after, what, 2 weeks, 2 weeks or 15 days. So we have to be able to think smartly how we create it so that in a couple weeks' time we can then implement and recreate it again. So that kind of, that sort of mentality has been really helpful in the way that we think about things right from the get-go.

So no swampy swampy.

No swampy swampy.

No swampy swampy.

And no, like, yeah, questions a couple months down the line, or how did we create that again? What was this where statement that we've all got?

I think one of the things I see people talking about is this idea that you can experiment and then lift and shift the code. And it's never worked in my view, in my experience, but the idea that you can experiment and then refactor, and actually if you experiment and somehow closer to the way the code needs to be refactored, then the refactoring is less, right?

Yeah, I mean, that was how we sold it, lift and shift, but I knew it was never gonna be that, but at least it's in the same technology and it's the same platform and you could take If they'd built views, they'll more or less work and it tells you what you're trying to do and things. So it was always going to be close, but it was never going to be pure lift and shift. And as we learned, there's so many ways to do things. It was just get the data in there to let them play and start something, or even just small reference data as well. It was always the problem, how do you load that, that 10-row table? How do you store that? And that was what we wanted to achieve early on. So we didn't have issues getting small things and/or big things.

Yeah, and I think the biggest thing as well in this onboarding or doing some stuff with the Scrum team, the data scientists, was that we got to see how they, what tools they used, how they actually ran like virtual environments and things like that. So like what is very much a simple kind of thing, like knowing kind of how they worked and knowing some of even just the tools to be able to analyze some of these datasets was actually where we've seen a lot of the value. So then as a team, we're kind of using all the same tools and we're all kind of working efficiently in the same way. So when we all run the same piece of code, it's all— well, for data scientists, we were doing everything relatively in our own environment, or very much personalized. So in terms of, I guess, it's— I guess what I'm looking at is around the sharing of stuff between the dev team. They were sharing stuff perfectly Right, you know, but as kind of data scientists, we weren't quite sharing in the same, same way, rather inefficiently. So just kind of—

and then probably also, you know, you're— I think you started adopting some DataOps behavior where you're checking code into a repository rather than saving it on your local personal directory. Now, yes, now, now, yeah, yeah, yeah, it's a final, final bit. Yeah, it's a journey, journey.

So these ad hoc Analysis are now in repos as opposed to some directory buried in the bottom of somewhere.

On your H drive, Liam personal test 2 prod do not delete.

So, and like now there's, and obviously when we put things in repos, they have to have READMEs, like they have that, like just things like that.

So it's all kind of little, little steps have kind of now Yeah, I think I liken it to, you know, when you're not a unicorn data scientist, that you're just a human that actually wants to do good analytics, these tools and techniques. And that adoption is really interesting. So, you know, if I look at DevOps and DataOps, right, kind of thing, what I'm seeing is a lot of kind of agile data and analytics teams now seem to be deploying their own infrastructure. They're not, you know, traditionally it was an IS job, and even if the data analytics team are in IS, they seem to do it without the traditional IS team, and I'm seeing that a lot. And one of the customers I'm working with at the moment, they have a very traditional release process and infrastructure process, so even though they have gone cloud, they still believe that the IS infrastructure team should be doing all the creation of the scripts to stand up and burn down infrastructure. But I work with multiple teams now where the data and analytics teams are actually doing that for both the code and their infrastructure. And I know that's something that you did, so why was that? Why was it something that you built? Was it because you had to, you wanted to, had to, wanted to?

Wanted to know what was going on so we could be masters of our own destiny. I don't want to say don't trust anyone else, but if we knew what was going on, then we could shape it how we want it. And I guess the cloud platform we chose, when you look at it, we were probably the most educated people on what we needed to do, and data analytics was maybe a specialty in it. It wasn't spinning up compute, it was all serverless, we were using as-a-service, and we knew, we knew what we wanted to do, but we just had to do it in a new way. So if we had to wait for someone else to do it, we'd be still waiting. So we just got on and did it, I guess. And it was, yeah, and we wanted to do it too, like learning new things. And it was, it's worked well, I think. And we've built a platform now, and within the platform we've built our own little area and when the time comes we can hand the platform over to the, to the responsible team and then we can manage our little, little area. But by doing it that way, it let us shape it as well and get, get what we want. So it was probably something you've taught us, Shane, is don't— control what you can control, and if you have to rely on other people, you're in trouble. And that's what I think we thought.

You may not be in trouble, but it's going to happen in my experience, right? So self-organizing means control of your own destiny. And the other thing for me is code's code. So, you know, as long as we're no longer buying hardware and shipping it from Singapore and racking it, you know, in a cupboard in our office, and then, you know, getting our CDs out and putting VM on and that kind of stuff, code's code, right? So as long as the platforms we choose, our servers are code, our data's code, everything's just code and we treat it as such, then then why not? Yeah.

I think everything's in code. 98% of it's in a build server. We've got a little bit that isn't. But apart from that, we've done pretty well, I think. And we've had to refactor. We had to move regions as well at one point. We moved from Australia to America. And we did that all in code. And it all works. So being able to make those decisions quickly and act on has been we would not be where we are if we had to have a wide group of people probably.

I know one of the challenges with the stuff that we're building at the moment in our startup, one of the challenges we have is I think I stole one of your techniques actually, and I started getting alerts from the cloud provider into my Slack channel of what's changing, and so it's like, holy Every morning I'm waking up and it's like, ah, shiny, cool. But that's part of the value and the problem, right, is actually figuring out how to manage that rate of change. You know, things that you go, damn, built that last month, wish I had known, you know, versus, oh, that's going to make us go a lot faster, but probably not this iteration, is a challenge, right?

Yeah, I mean, we use a lot of things that are in beta still to our own detriment. Probably as well. So it's just, that's just how it is. I didn't realize how much was, but it's just we need them, so we use them, and that's just decision we've taken. And it's not, if we have to refactor, we will, and that's, but you may not.

Yeah. All right, just looking at time, might close it out. I've got a question for Liam because this is one of my favorite ones. So I think I was, I replied to somebody, I cheekily replied to somebody's LinkedIn post the other day when they were talking about the top analytical routines that most data scientists run, and I said to him, you forgot to mention number 1's GROUPBY. So I come back to the point you made slightly earlier, which was actually in terms of the maturity of what you're doing, yes, machine learning and vision analytics and natural language is all great and high value, but actually at the beginning, you know, some simple, using some data with some simple techniques to start start that journey is actually probably what you're doing. Now, paraphrased you there, did I get it right or?

Yes, well yeah, the old group by count, count of nulls, all that sort of stuff. Obviously understanding the data first and obviously I think you showed me a thing around about that Pandas profiling thing, how they like, they're sort of—

Descriptives, right?

Descriptives.

Yep. Like just initially looking at the data data first to see what's missing and what's not. Obviously that was a big part of it. And there are all these little services as well that, you know, you don't need to know a large amount of data science to be able to use some of Google's natural language stuff. But I guess being able to understand some of the bias between some of it and understanding where its pitfalls can be is kind of, well, some of the stuff that we've been investigating.

But yeah, I guess the old group buy count Start off with the easy stuff and then build up to the complexity, right, where you have value in it. But I think, you know, with the buzzwords out there, I mean, now that big data's dead and, you know, MapR's gone bankrupt and been bought out, so, you know, the Hadoop, I think big data Hadoop world's died, but we're now on the world of AI, you know, AI for everything.

But when possible, I try to avoid the words machine learning and AI, because they do, they are those buzzwords that do get confused and misinterpreted like some of the agile words, right?

Yeah, I know them.

So we try to avoid them and use detail when possible, if that kind of answers the question.

That's a good answer. Excellent. All right, well, I think we might close it out there. So thank you, and thanks for letting me record in the pub.

That's all right.

I do want to apologise to our listeners for the choice of music. It was not my, Hamish's, or Liam's choice of background music, but it was just the location we're at. Excellent, right, we'll catch you all later.

Thanks for having us, Shane.

Ah, no problem.