The pattern of writing a data book with Shane Gibson and Ramona C Truta - Episode #64

Welcome to the Agile Data Podcast, where we talk about the merging of agile product and data patterns in a simply magical way. My first book, An Agile Data Guide to Information Product Canvas, is published on Amazon. If you hear your stakeholders say, "Hey, that's not what I wanted," and then you hear your team reply with, but that's what you asked for, then you probably have a problem capturing data requirements. The Information Product Canvas is a pattern template that is a quick and simple way to solve that problem. Head over to agiledataguides.com and click on the green book link to buy a copy now. Welcome to the Agile Data Podcast. I'm Shane Gibson.

And I'm Ramona Citruta.

Hey, Ramona, thank you for coming on the podcast today. Today we have something slightly different because today I'm going to be the guest and you're going to be the host. So this is going to be a really interesting experience for me. But before we do that, why don't you give the audience a bit of background about yourself?

Okay. I have started in data since ever. I joke that I came on this earth doing data, and I'm going to explain. I've always been the kid who arranged all the fundraising, did all the financials. I was the trusted kid who really did every year all of that. And so it was data, right? Not Excel spreadsheets, but with pen and paper. And I loved math and always dreamed of becoming a math teacher. At that time in Romania, we had to pass an admittance exam and I went to fill in the forms and they had 3 separate options. One was in computer science, one was math, and the other one was applied math. And the idea is that everybody would write the same exam and based on the mark that you get, you'll be distributed in your section. So as I was filling in that form, a friend of mine told me, why don't you put computer science first? Because you can do with computer science everything that you can do with math and some. Of course I wrote that first. I got a very high mark. I got into the computer science program, clueless as to what that meant. My first reaction to the first computer science class was this: x x 1. Now for a mathematician like me, I was dead in the water. I just, let's just say I had a very terrible first computer science teacher and I struggled. I struggled with the computer science for a while. And luckily for me, I excelled at math. So that really counterbalanced and all the logic and all those other courses. Long story short, I get to the databases course and coincidentally at the same time with the probability and statistics. So that saved me. Fast forward. My first job out of university was working with data and FoxPro. I worked for high school and I was preparing everything from design to implementation. The next job for a multinational, it was with Progress, another database job. And my first project that I was working on was doing black box testing. So this application that we're developing was for a Dutch company. And everything was in Dutch. I had to use two separate dictionaries because there was no Romanian to Dutch translation. I had to use English as an intermediary. I love that job and they advanced me very quickly, put me in charge for the project. And that's the time when I got to a job in academia, which was actually my dream. And a funny tidbit, I had to do a lecture, part of the series of interviews and my lecture was in data modeling. I always remember this and I got to teach that. And then I met my future reason to move to Canada. I got a scholarship to come and do a PhD program at University of Toronto. And during that time I was in the database group. So I continued that and my research was in XML, XQuery. So we did that kind of research at that time. And I started teaching the database courses at the computer science department at University of Toronto. And I've got to teach other courses, software engineering and other courses to make up the schedule. I've been doing that for a while and then I stopped teaching and moved to a different university to do research. And I was the data person and that is where I really got to practice all the things that I've been teaching and more. And to this day, I guess that was my absolute favorite job. Fast forward to today, what I am now is a facilitator of knowledge. This is what I'm calling myself these days. I just love to do that in the sense that I take the tech talk, like really what the technical people talk and translate it into business talk and vice versa. And I talk a lot of strategy, especially in the data and AI era that we live. There's a lot of translation that is needed to be done. You probably have witnessed this yourself. A lot of people talk like they know everything, but it's just what I call the foam on the top of the cappuccino. And I'm the person that goes down to the caffeine. So, I removed the fluff, I removed the foam, and I speak caffeine, like really strong espresso caffeine.

As a coffee lover, I'm there with you. So, that idea of being a translator, I think we see people fall into that role over time. And it's typically people, we've talked about our ages before we started recording, but people of our generation, our age, because we've spent years observing and learning and training our brains to recognize patterns and be able to go, oh, actually, I can describe that quite clearly because I have all these frameworks and patterns that I can anchor to. I have these examples and I can bring those in so that I can explain something that's complex in a relatively simple way to somebody else who's never dealt with that complexity. And I think that is a really important role, especially now with all the AI stuff coming, that ability to be able to provide that role is gonna become far more valuable than it's ever been.

Absolutely, Shane. And I'm really glad that you brought up our conversation prior to recording because what we said, we've lived several decades and we've collected a lot of data points and data. We have these massive data sets and now we can build those knowledge graphs of everything that we have accumulated during this time. And while the New technologies may look new, but they are built on similar principles and we have a system thinking, and then we just apply all those frameworks. And I think this is where we are really valuable because we have all this body of knowledge that we've accumulated and we've built and we shaped our minds.

Hopefully it's true because I'm really bad with a hammer and a screwdriver or any of those kind of, so I'm gonna have to rely on the last decade of my working life still being knowledge, not of hard work. Okay, like I said, we're gonna flip the script. You're gonna be the host, I'm gonna be the guest.

Yeah, so first of all, Shane, huge congrats and I applaud you for finishing up the book and putting it out there and I hope the book does well.

Lots of people have asked me why did I decide to write a book and it was the scratching itch. It was just something that I decided I wanted to do. And so yes, there's lots of knowledge in there I hope people adopt. I hope it's successful. Somebody told me really early on, you don't make money out of a book and looking at how much you get paid, You don't. And so for me, it was about scratching an itch. And the other thing people said was it's incredibly hard. Like lots of people start and never finish. And it's been a few years and now I understand why. It's really interesting. You don't really understand how hard it is. It doesn't seem that hard, like to just write a book doesn't seem that daunting at the beginning.

Especially with ChatGPT, right? You just prompt it there and then the book writes itself. Yourself, right? Yes.

Yeah.

And we know that there are books like that. Tell us a little bit about, first of all, the title and how you decided on this particular book. I know it's all about patterns and I shared with you earlier that ever since I've met you and I've known you, it was all pattern this, pattern that, pattern the other, and come on, what, why does everything has to be a pattern with you? And it took me a while to understand what was a pattern to you. So take us to this journey, if you will.

Yeah. So let's start off with patterns, right? There's a book that was written many years ago, A Pattern Language, and it was about patterns for building houses. And the concept of the pattern really is there's a common problem and there's a common solution that fixes that problem most of the time. And therefore, if you can articulate what that solution is to that problem, it becomes a pattern. If we think about a house, when we want to build a house, there's inherent set of patterns in that house. If I want to build a lounge, I am typically going to have a room that is a certain minimum size. It won't be a cupboard. It'll be bigger than that. It will typically have, depending where I live, lots of windows and light. It'll have a place for couches and seats. It probably in the modern world will have a place for a television. It may have a heating source like a fire. If I compare the design of that room to a bathroom, a bathroom's typically gonna have one window, and that window's probably gonna be opaque because you don't want people peering in. If I live in a sunny place, I'm probably gonna have a porch. I'm gonna have an open space outside where I can sit and it has some form of shelter from the sun, but it's open, or it has a series of windows I can open. Each one of those is a pattern. It's a solution to where I want to spend most of my time during the day, where I want to go clean myself, where I want to sit when it's sunny and fresh. And that's what the book was all about, was describing these patterns for architecture. And if we think about that one more step, is if we were going to build a house, 9 times out of 10, we'd do architectural drawings. We'd do a blueprint. We would sketch some details that make us think about the layout and the patterns we're going to apply. And also give us something to follow and iterate as we learn some more things. And so for me, that was really intriguing. It was intriguing because a few decades ago I had a consulting company. So the usual 5 to 20 people data and analytics consulting company. And I used to get really frustrated that whenever we went into a new customer, we seemed to be reinventing the wheel. We're like, oh, let's start from scratch. And I saw a massive amount of waste and inefficiency when we did that. And so my natural reaction was to go to methodologies. Yeah, oh, we'll write a methodology, we'll write this book of rules, and everybody would follow the rules. It was a freaking epic failure. Nobody followed the rules. The rules didn't work half the time because every context was slightly different. The amount of money we spent trying to write the methodology was waste compared to how much we got out of it. And so I got really frustrated, and then I lucked into this thing called Agile, which completely changes the way you think. Aligned with that came this idea of patterns, of small chunks of proven solutions for a common problem. And then what we call anti-patterns, the times that we know that pattern actually introduces more waste than the problems it solved. So when I first started to write, I thought about what I wanted to write, because I had many goes at starting that failed. I sat back and went, oh, I'll just write this big book of patterns for data. And that's actually the first thing I tried to do. It was an epic failure in terms of the outcome, but it was brilliant in terms of the learning that helped me finally finish a book.

Oh, okay. Why do you think that happened? What exactly made that failure happen? Or what have you learned from that aside from shrinking the knowledge and trying to fit something? Did it change your perspective? Or your approach, what influenced the most and what made you change, what you took with you and you carry with you in the future?

So I think for me, it was understanding the way that I'm driven. Like I said, I had a whole bunch of false starts many years ago, over many years. It was like, okay, I want to write a book. It's a niche. And then I start to do something and nothing would happen. And one of the things everybody tells you to do is write the outline of the book. You know, write the table of contents. And for some people, that is a good approach. For me, it was an epic failure because what I kept doing was I kept rewriting the table of contents. I'd write the table of contents, I go, yep, and I'd write 2 paragraphs of actual content and I go, oh, that doesn't fit the structure of the book. So I'd go and seriously waste more time restructuring the book. And after a while, I realized that was an anti-pattern for me, that create the table of contents. The other problem I had was I would write when I felt like it, and then I would not write. And that consistency was not a good thing. So I decided that what I needed to do was a forcing function that would make me write consistently and not worry about the content yet. And so when I do forcing functions, what works for me is make them public. So I said, I'm gonna write 60,000 words in 6 months. So that's around about 10,000 words a month. I am going to, on LinkedIn, publish my word count. As a recurring theme so that I could hold this metric to myself of how many words I wrote. I am not going to care about the quality of the words. I'm not going to care about what the words are. I am just going to prove that in 6 months I could write 60,000 words. And I did that and I used Substack and I created the Substack site called the Agile Data Big Book of Ways of Working. And that was just 60,000 words. Now, what was interesting about that is it forced me to research. So I would write about a pattern for, say, team design. And I'd go, yep, okay, this is the way I design a team. Or actually, no, the example I used is agility. Because I'd go, I can't describe it. It's one of those things you go, what's agile when you feel like it? And I was like, that's not a great piece of content. So I started doing research and then I found this research around fixed mindset and growth mindset and a book by Carol Dweck. All around that. And what I could do by doing that research and doing that reading is I could see the patterns that were anchoring me and where they might have come from. Because I can't remember where I read them. I read lots of books, I watched lots of things, I talked to lots of people, I experimented, but I couldn't see the genesis of where the idea came from. And that actually made my content better in my view, because now I, I could think about it more. I could rewrite and write some stuff. And for me, writing those 60,000 words, was 6 months. It gave me a whole lot of process that I've used from then on, but also gave me a whole lot of content ideas and a whole lot of research. So that was the first step, 60,000 words of absolute crap to prove I could write 60,000 words.

And I think that's when we first met, right around that time. And I remember I was supposed to contribute to some Substack and You said, oh, this pattern, and you gave it a name. And I read the thing and I said, oh, I know this thing. I just didn't know that it has a name. And I think in the perfectionist in me was upset that how come there is a thing that has a name and I do the thing, but I don't know the name. So this is where my frustration with everything pattern and name came to some context because you know, that perfectionist Why don't you know this thing? And I started beating myself up. Anyways, that, that's an aside, but it sounds to me like even in the first iteration where you were trying to come up with that table of contents, it really sounds to me like you are trying to figure out a pattern in the knowledge that you wanted. It's meta the way I see it, that you try to approach everything from a pattern point of view, including the thing that describes the pattern. I don't know. In my mind, it looks like that.

So I think now in hindsight, after a couple of years, I think that's what I was doing, but I didn't know it at the time. And actually, again, so forcing functions are really important for me. Otherwise, I just won't get stuff done. I was the kid at school that was finishing, as you say, at 2:00 AM in the morning because he hadn't done it for the last 4 weeks. And that really hasn't changed in the way I work. One of the things that was intriguing me was, why is it so hard to write a book? And one of the things I've got to now is actually the first book's hard, but the second book's harder. And so I've got a challenge on, can I write a second book in 6 months using the patterns that I've created for this book? So we'll see how that one goes. I don't think linearly. I don't think in a structured way. I think in chaos and randomness, and that's just the way I've always worked. And I am constantly distracted. So I had to find a writing technique that worked for that. And then I had to learn some new skills because I lasted 6 months at university when I was a kid, and then I left and went to work. So I'd never really learned the research skills that you get from a good university. So what I found was I could write, but I couldn't justify what I was writing. And therefore, I had to learn this whole new pattern of research. Whereas for you, I think you understand research intrinsically because you've done it all your life. And so you had that foundational piece that I didn't have. I think the second thing is over the last couple of years, I tend to prefer to work with other people. I find it more invigorating. I find it more fun. I find having somebody else that I'm working with gives me those deadlines to force me to do that work. It's a forcing function. And I have tried multiple times to write this book with many different people and every one of them was an abject failure. And it's really interesting. It's like, why? Is it me? Is it the book? Actually, Collaborating on a book is actually harder than writing it by yourself, I think. So I've got to resign myself now that every other book I'm going to write is just going to be me. But having said that, in the book that I did publish, there were a whole lot of example canvases that people generously spent their time creating without me. They've gone with the book. So actually the book is a collaboration. It just wasn't co-written in the typical way you would co-write a book, which again is an intriguing pattern.

You just, I was just gonna comment that, that eventually you found your pattern to write this book, something that worked for you and the collaborators. And we have a few friends who wrote books and probably they are agreeing with you on the side of having co-authors and collaborators. And it's hard in general when you work with somebody and having different styles and it's It's not for everyone and it's a different set of skills that is required and always having to push towards your own issues, the other person's issues.

I think the other thing is I'm highly opinionated and I had a certain vision for the book, like the layout. The book is about the Information Product Canvas, and that is an iteration of the Business Canvas. And I love the Business Canvas book when it came out, just the style, the form and the function of it. So this book is an iteration of that style. It is more coffee table than it is textbook. And I think if I ended up with a true co-author, there would have been a hell of a lot of tension because I don't think I'd be willing to give up the structure of that book. And in fact, the early feedback I got on the structure of that book was that it was crap. And it's going to be really interesting now that it's published to see whether that's still true. But it's the book I wanted to write. And like I said, I wrote it to scratch an itch.

Did you succeed? Hard scratch? Yeah.

Yeah. It was definitely a hard scratch. Yeah. I like the book more than I thought I would. Which for me is a big compliment because normally I'm doing something and I hate it.

Okay. So then it's a success. You achieved what you set out to do. Yes. And you were happy with the result.

Yeah. But a couple of years late.

But let's go back a little bit in the process and see what other patterns we are identifying. You say several years later, but there was a purpose, or since you studied that fixed versus growth mindset, you had to transition towards more growth mindset than the fixed one. So that gave you plenty of opportunity. So just consider how much growth there is and how much you got out of that process on its own to achieve that kind of shift, because that is huge. And especially as we grow older, if we are a fixed mindset, then the older we are, the older we get, the much harder it is to shift to the other side. So I wanna congratulate you on that and give yourself grace for it. It's a huge achievement.

And I think, yeah, the fact that actually I finished it, cause you know, I do tend to start some things and not finish them on a regular basis.

Shane, let's admit that's a pattern that I don't know anyone who doesn't follow it. Let's put it that way. And especially now with AI, it's so easy to experiment and, oh, I have a weekend. Let me vibe code something. I know your opinion. And I did about that. I read what you wrote because everything is so easy and then you start something and then the mind goes to the next thing and the next thing. And at the end of the weekend, okay, what do I have to show for? And this is something that I bring up more and wanna talk about the new paradigm or pattern to use your preferred term of learning how to learn. There's a lot of shift in how we learn. Now in this context, and for us old-timers, we have a lot of knowledge as we established at the beginning. And now with the vibe coding and all these tools that do a lot for us, we immediately can make connections. Oh, it's doing this, even though it's something that we never saw before. Code, we never wanna learn and do it ourselves, but we have anchors and there are things that we have knowledge about. We may not know syntax and stuff, but we understand what's happening. And then we can go and ask in a chatbot, hey, tell me more about this and teach me more about this. And then check my knowledge. I do that for myself. It's a lot of fun. And I don't know that I've ever been this enthusiastic about learning new things as I am now.

We've got access to a knowledge base in a way we've never had before. I'm old enough to remember the Encyclopedia Britannica as the thing you used to have to go and look up. When you wanted to understand what a thing was. You used to have to go to the library at school and get a book out and read it. And everything's changed. I think there's some interesting concepts out there now that with the advent of the LLMs and the GenAI and the access to that knowledge, that curators of that knowledge, humans that actually tell a story based on that knowledge in a different way, will actually become the more successful people. Having the knowledge is no longer good enough. Because that knowledge is now available at a click of a button, type of a key. It's the way you tell the story that becomes intriguing. So we're going back to thousands of years ago to storytellers being the most valuable part of the tribe. And that's intriguing. So time will tell.

Intriguing how?

Because it goes back to an old pattern. It goes back to the storytellers and the way they told the story becoming more important than the holding of the knowledge. The articulation, the ability to have somebody else understand what you understand, to coach and mentor and teach. Because the knowledge is just available to anyone.

And it's just the way we transfer that knowledge. Going back to what I was saying at the beginning, that I'm a facilitator of knowledge and I'm a visual thinker and a systems thinker and all sorts of other things. And what I observed, a pattern, I observed this pattern that I have conversation with many people, just as you said, right? You talk to a lot of people. I read a lot. I see a lot. I consume a lot of knowledge. And suddenly it's all that chaos gets into an equilibrium and I get a visual or a simple sentence, but I get a picture and that picture can be described in so many words and I can use it. It's a pattern or a framework on its own that I keep using and I keep finding usage for it, like the cappuccino thing or another one I came with. Was the puzzle versus Lego. In this highly siloed world we live, everybody's working on their own tiny piece of puzzle, and then they expect that they all get glued together somehow, or everything fits, and expect to work like Lego, which no matter which Lego you buy, which Lego set, they will fit on top of another and you can build something. But That's not the reality. You know how Joe encourages everyone to write an article explaining why they write or their process. So I was 13 and in Romania we had to give an exam at the end of grade 8 to go to high school. So we had to decide what kind of high school we wanna go, et cetera. And at 13, I remember vividly I had this conversation in the schoolyard with my Romanian literature teacher, and she asked me, so Ramona, what are you going to do? Storytelling or math? I was exceptionally good at storytelling. I know how that comes across, but it was effortless for me. And I would go to competitions, Olympiads. I could never qualify for math to higher levels, but for storytelling, I would just go and I would always win. So I had to make that decision. And what hold me back for going onto the humanity side of training was the fact that I didn't feel good enough for a foreign language. So my first foreign language that I had to get in school was Russian, then French, but I had such a terrible teacher that we were clueless. With French. I've learned French going to high school, so I picked it up. English when I went to university. So I go into the sciences, I did math and physics. And I remember years later in grade 11, I'm in this class with these kids, everybody who was anybody in my little town. And we have the Romanian literature exam, grade 11, and we are given a fragment of one of the books that we've been studying that trimester and to write an essay. And when the teacher brings back the exams, she acknowledges me in front of the whole class. She highlights my work and my essays and my storytelling and all of that. And I thought, huh, how interesting, because I was never acknowledged in that way for that particular subject. And it was the only time she ever gave me the 10 out of 10 mark. That stuck with me. And here I am decades later finding myself in that passion of telling stories again and want to tell stories about so many things.

I remember years ago, I used to work for SAS, the software company, statistical company, and I was in pre-sales. So I wasn't on the implementation side, but I spent a lot of time with, back then they were called data miners. And one of the techniques they had that still sticks with me was they would go and build an analytical model using a neural net. And they'd do it with a bunch of different stats techniques. Often it was marketing related. So they'd look at the lift chart and they'd go, that model with those parameters is the best fit for solving that business problem. And sometimes the neural net would be the one that wins. The problem was you could never explain to the customer what the hell it was doing because How does it work? Well, basically we push all this data in, we set these 3 flags, we let it run, and it came back with the best answer. So what they used to do was they used to produce a decision tree model at the same time, and they used to use the decision tree to explain to the customer what's happening. And they would say, this is not the model, but the model is doing something like this. It's looking at customer and it's looking at length of time they've been with us and then total spend and then number of contact center calls to come up with these clusters. And although it wasn't the actual model, it was able to tell the story that the model was probably doing. And so for me, that's always stuck with me is that we have to be able to take these technical things we do and explain them by just telling stories. And when you're writing a book, for me, it became actually, how do I tell a story? Because I know the techniques, the patterns that I was using, because I've used them for a decade with lots of people. I knew they worked, but there's this really weird thing that when you're forced to write, actually it helps enhance your understanding of what you know. And so on this podcast, what I do now for this year, I will say to somebody who's coming on as a guest, write an article about the pattern we're going to talk about. Not because I need to see it, but that art of writing, that process of writing that article about what you think you know. Will make you know it so much better. And for me, that was something that I didn't expect because I, my whole life has been talking and waving hands and drawing pictures compared to math or writing. And so moving into the skill that I didn't have, and I still don't think I have it very well, but it helped me articulate and understand what I thought I knew a lot more because once you've written it and you go, that's just pants. I don't understand what I wrote. How does anybody else do it? I think that is actually part of the value of writing. So again, something new for me.

You've explained that really well because we have so much in here, right? So much as we've accumulated. And I see this in all the conversation, especially with tech people, and they immediately go to tech jargon. And what I want is I want to hear a story that is tech agnostic. Can you distill what you're saying without all the jargon. And I've been in meetings, sometimes we were in those meetings together and at all times I had to figure out a translation in my head. Oh, this person is talking about that and this person is talking about the other thing, but I think they talk about the same thing, but they can't figure out that they're talking about the same thing because they use their own particular jargon. So that was a big shift for me, understanding that people around me are talking pretty much the same thing. They just don't use the same vocabulary or they're very particular and very opinionated on their respective jargons. That's why I became so passionate. I've always been a teacher. I've taught thousands of students and had to explain and distill and all of that. But it gives me so much more passion right now, realizing how people just touch the surface of things and don't know what's underneath, but they would like to know. And they just need a different way of explaining that without all the jargon and all the technical details. You just wanna understand what's in that box at a level that allows you to have a conversation with somebody else or a convincing conversation.

And one of the interesting things for me was, so I started off with 60,000 words and I looked and said, actually, if I documented every pattern that I think I know, it will be 600,000, 6 million, and it's never going to happen. I then picked one of those areas, which was the canvas, and I started writing the book and it got too big again. Because a pattern, especially in the data world, doesn't live on its own. So the canvas is a pattern, but actually there's a workshopping pattern, there's the who does what pattern, the core business event model pattern's already in there, prioritization, personas, there were all these other patterns that it used, it leveraged. And so I ended up again with a book that was too big and I had to stop myself and refocus. And then the other thing was whenever I used a term, I started to go into too much detail. For me, it was like, oh, I'm going to go and explain it to the nth degree. And again, that made the too big. I was going, if I keep writing at that level, I'm never going to finish. And so what I decided was to write a book for people that were inquisitive. So it tells you what you need to know, but actually you still have to go and do more work. And it's really interesting, as part of our startup, we have people that are in the middle of their degrees from around the world come and do work with us for 3 months. And I remember I had one student and we give them a bunch of choices of what they can work on. And he said, he wants to get into product analytics. And I said, okay, you've got two choices. You can do product analytics or marketing analytics for a product. You need to pick which one that you're going to do. And he goes, what's the difference? And I went, actually, your first task is to go and give me back one page that describes the difference between product analytics and marketing analytics. And I waited and I waited and a week later I went, where is it? And he goes, I couldn't find anything. And 10 seconds later, being grumpy old man that I am, I'd Googled it. I found 20 references and I sent it back to them.

Them.

But what that meant was he wasn't interested. He wasn't inquisitive. He was only doing this to pass a tick in the box. He didn't actually want to learn. And so I think again, that's part of it is that books are there to help you learn the next thing you need to learn. But that second step, going from the 60,000 words to what I thought was very focused and then starting to write and then realize, nope, it's still too big, forced me to go to this third iteration, which was, what am I not going to write? And that was more important. The constraints-based model of, I am not gonna write about these things, was the last thing that I had to understand in my writing process. So start really big, lots of stickies, complete chaos. Find something that looks like it coalesced into a great story that I could tell that was different to every other story that I'd read. Start to write that story and realize it's still too big. There's too many tendrils. The graph's got too many nodes. And then a constraints-based model to say, what am I not gonna do? And then that was the process that let me get to 193 pages in a couple of years. And the test, like you said, is I think I have a pattern to write the next book. I know what the subject is. Can I reuse that writing process that I think I have to do 193 pages in 6 months? That's going to be the test for me because that's when there's a pattern, not just a bunch of ad hocness.

So Shane, during the process of writing this book, have you uncovered some new pattern that you didn't know before, something that really excited you so much that you wanted to include, and maybe something that is one of your preferred patterns right now?

Yeah, there's a couple. So, one of the things that happened when I was writing the book was I had created a course for the canvas, which I used to teach privately to organizations, and I made public on Maven. And what I didn't realize was running that course helped me refine the content and it also helped me get feedback on that content because effectively I was practicing delivering that content time and time again. So what I'm going to do next time is I'm going to write the course first and then write the book. And the second thing is I'm going to focus a lot more on the questions that have been asked by the course because I didn't record them this time. And what I found was actually the questions that I get asked is the content the book should focus on. So that's the first thing in terms of the writing process, create that course, run it, iterate the content on the course as I'm iterating the content of the book, actually document or record the questions I get asked by the students because that's telling me what I'm not articulating. The second part was the example canvases in the book have been done by a whole lot of collaborators based on projects they've done. And I hadn't written the book, so I had nothing to give them and they hadn't been on the course. So what I did was I took them through the canvases a quick half hour, here's how you fill it. Out and then let them go off on their own and do their own canvas. And that was intriguing because they did it in ways I didn't expect, but there were also patterns across every collaborator. So for example, in the canvas, there's a vision statement, and the last part of the vision statement is what's called unlike. So if we didn't build this information product, if we didn't deliver this data and information to the organization, what would happen? What's the alternative negative thing that would happen? And pretty much for every collaborator who worked on their own, the answer was manual process using Excel.

You're kidding me.

No, because every project they worked on, because they're all practitioners in the data domain, that was what they were replacing. And it made me think, we talk about the value of data and information, and we've talked about that a lot lately. We talk about the value of data teams. We talk about writing business cases and justification for this work to be done. But actually, maybe we're overthinking it because really the alternative is, do you want to run that process in Excel or do you not? Because that's actually the thing we're replacing 9 times out of 10. And that was intriguing for me. That spawned a whole lot of thinking about value of data in an organization. Are we replacing a process that's inefficient and do people really care? So that was another kind of surprise learning for me as part of that process.

So really, what do you do with that? So which answer did you settle on? If you ask these questions, right? So are we over-exaggerating things or are we overemphasizing on the value that we bring? How do you settle seeing this pattern emerging from your collaborators and those are the people at the ground level, right, ground zero, they are doing this. How did you settle that pattern emerging?

I haven't answered it. It started me thinking, it started me thinking data teams are like a finance team. Data teams are like a finance team without the mandate. A finance team, a bunch of financial people in the financial part of the organization, are a shared service, and they're mandated. You can't typically run an organization without a chief financial officer, without people doing the accounts and the general ledger and producing profit and loss and balance sheet statements. They don't articulate their value to the organization. They don't go, would you like to have a balance sheet and a profit and loss? But for some reason, data people do. We're a shared service, and then we have to go and justify our existence. And that seems like an oxymoron to me. It's intriguing. Maybe the answer is actually we do get embedded with the operational team. Teams, we no longer become a centralized shared service, we become decentralized. I don't know, there's lots of questions there. And when we do a data team design book, the guide for that will probably, I'll have to answer them at that stage. That's not the next book, by the way, that one's a hard one. Yeah, again, the good thing about it is that as you write, you learn, because you're thinking, you're asking questions, and then you're researching and exploring and collaborating. And so just the process itself is invigorating.

This was a really perplexing pattern to emerge, right? I know we always joke that, oh, Excel will never die and Excel is all you need, right? So not attention and transformers, Excel is all you need. And to actually have this end result of all this research and the work that you and your collaborators have done, you validated that which we took as not empirical, but we, as anecdotal information. So, you actually proved that is a fact. It's steeped in reality. It's really fascinating.

And these are people that are professionals and typically consultants that get paid to do that work. An organization has paid them or their teams or their companies to replace a manual Excel spreadsheet process. With something that's quite costly because data teams are costly. Intriguing. Yeah.

Very. And so in this context, Shane, you just proved that data teams are a cost center, essentially. So it's not the conclusion that you would have hoped to arrive at. But now we have AI and we need data teams to put the data in order so we can consume all that money to generate AI to fix some something that we can do in manual Excel? Is that what you're saying?

I'm not convinced that AI, the GenAI LLMs and the new way of working with those tools actually equates to us structuring our data more. I have a hypothesis that I haven't got around to testing that actually will structure our data less, but that's probably a different and very longer podcast.

Okay. So maybe we're looking at it from different angles. Let's clarify this a little bit because structuring or cleaning the data, those are different aspects of it. So for any data to be relevant and to get patterns, good patterns out of it, so we can learn something from it, it'll have to be better than just plain garbage.

I'm not sure it does.

Please elaborate.

So this is unstructured thinking. I haven't gone through and done the work, the research, the experimentation to get the core patterns out. So I'm very early in my thought process. But the LLMs have been trained on pattern recognition of large text repositories. That's what they do. So why do we think that our data factories, our way of working where we put all these processes in to change the structure and the context and the content of data, increase the quality? Is actually a better way of feeding those models than just giving it the chaos and asking it to find the pattern. That's my underlying hypothesis, and I don't know the answer to it.

As you said, this should be a podcast on its own. And Mr. Snarky McSnarky came out to play. Oh yeah, I was wondering when he will come out to play.

I think we have a whole new set of tools that have a whole new set of patterns, and there's a real risk that we're trying to apply legacy patterns that haven't worked for us anyway into this new world. And I just think that's dangerous. So I don't know. Again, we'll find out in the next year or two, because there's things accelerating so fast.

We are talking about data in general and how it's shaped and all of that. But in the context of a business, that data is pretty structured, or we want to believe that it is, right? And all the documentation, everything that that we can think it's digitized already or about to be digitized, it is in some shape and form better than how we imagine everything that was scraped off the internet. It's a different kind of data volume sets, whatnot. And when we talk in the context of the business and the information that is pertained inside the business and the rest of it.

So let me give you a concrete example that keeps coming back to my head. Again, I don't come from the library services domain. And so whenever I hear ontology and taxonomy, I get a whole bunch of technical words that I don't quite understand. That's not true. I kind of understand them more now because I had one on one of the podcasts to explain to me what the hell all those things mean. But they're about putting a form of structure against some things, a classification to a degree. And my question is that a better pattern for using that type of data with GenAI and LLMs versus just chaos tagging? If I went in and just tagged the snot out of every piece of data that had any kind of value with any tag that it might be, and then put that chaos into the LLM, will I get a better result? And that's my question is that actually does more unstructured chaos to let it find the patterns, is that better For example, it's the difference between a k-means where I just throw all the data at it and say, tell me 4 clusters, versus a SQL statement where I put a if-then-else in to say, here's 4 clusters. I'm just intrigued on the difference of those patterns right now. And I think we are applying the tools that we know to a problem that we haven't proven. That's my view at the moment.

Well, I guess time will tell. It's the entire non-deterministic nature of it. And there have been so many papers coming out and there are examples of one-shot learning when it saw it once and that is what it learned as opposed to other things that have seen it many times and it didn't learn. So that is an issue because you don't know in the end what it'll learn, what it'll be the end result. I'm on the structure part of the equation and this is just just how my brain works, and that is what makes sense to me. But I understand, and I know there is order and structure in chaos. You just have to find it, and that requires a different technique, different set of patterns.

Yeah, and Juan Siqueira challenged that versus a non-deterministic model in a really interesting way. What he said was, because we know the machine is not deterministic, because we know we will ask it a question and get a different answer, we don't trust because we expect the machine to be 100% right. However, when we ask a human, when we ask a data analyst to analyze that data and give us the answer, we don't expect 100% accuracy. We don't ask to see their workings. We don't ask them to prove it. We trust that they know what to do with their job. So there's this dichotomy between not trusting a machine but trusting a human, whereas actually a human is probably far more non-deterministic than a machine to a degree. And an example today on the Practical Data Modeling Discord that I posted was I saw somebody post something on LinkedIn that said the technical tools that a data scientist use is Excel and Tableau. Like bollocks. The term data scientist should be the clue that there's some steady science involved. And they actually listed, they listed the stats skills that a data scientist needed. So they said the skills you need is stats and they listed a whole lot of stats capability and patterns you needed. And then they said the tools is Excel and Tableau. And so I can look at that and I can go, that's a disconnect. That doesn't ring true to me. I don't believe that based on my experience, but how many other people would? And I think we seem to think that humans are infallible and we should trust ourselves where actually we can't, yet we won't trust the machine because we know it hallucinates. I'm intrigued again by that dichotomy.

I don't know who created that, and there were some other issues there if you went to the right most column with the gen AI engineer, and then he, there were some other typos there. Probably it was generated, who knows. But back to this idea of humans being more non-deterministic and the machines being non-deterministic. So why do we trust humans more than machines? I've heard Cassie talking about this. She wrote several articles on this idea that It's not that it's more non-deterministic, but there are a lot more correct answers. To go back to that, which one do we choose better out of all these good answers, correct answers, which one is more correct? So they are all correct. So how do you choose? How do you make decisions? And especially from a leadership standpoint, what do you do with all of that? You have to make decisions. You have. To entrust in the end in something and anchor and ground. What is the ground truth? What do you do? And it's an interesting point with humans. We have such a large spectrum. You put two different people with the same title next to each other and you will need a translator in between them.

We just gotta take the arguments we have about what is a semantic layer. Oh my God. Our semantic arguments on semantics.

You were in that meeting?

Yes, I think so. And then you got the what's a data product argument.

And then in that meeting with the semantics, that, okay, what exactly are you talking about? From where I'm coming from, the academic side and the research, this is what I think of semantics. What are you talking about? And then it was complete silence, and I didn't know what to do with that silence.

But because you can speak multiple languages, You are doing a form of semantic mapping to be able to take Romanian and make it French or English or Russian. Your brain has been wired to that mapping. And one of the things that I look back on my life and go, I wish I learned a language at 5 years old, a second language, because I think fundamentally that would wire your brain. Yeah, maybe 3. I don't know what the age is for the brain wiring, but to learn a second language fluently at a really young age, I think would wire your brain completely different and actually allow your ability to semantically map different patterns in the world to happen a lot faster than if you hadn't learned multiple languages. I'm intrigued by that one.

My daughter, she is so interested in languages. Well, languages, but, and more into symbols and alphabets and scripts. So she had taught herself more than 10 scripts. She wanted to call herself a poly-alphabet, so she's now a poly-alphabet and she explains it scientifically. So she did all this research prior to ChatGPT on her own, and she would tell me that sound and that phonetic and this and that and wow. So it's all very systematic and she found the rules, patterns. I know I kept throwing patterns at you as an inside joke between us, but now I'm serious. She uncovered patterns and she could figure out connections between a pattern in some script and patterns in the other script and how they translate. I'm realizing this in real time in my conversation with you right now.

Oh, and she's multimodal, right? Because she's doing it with both tics. And audio. So she's got the ability to be multimodal.

Yeah, and she really wants to stay away from AI. And Joe talks about his own kids and how, no, they insist on not being influenced by AI and what is this doing and that. And we on the other side, we embrace it and we see how much help we get. And that's another pattern emerging.

I think because we've had to do the hard work ourselves, therefore I think we can leverage at all. I wonder if you grow up with knowing nothing other than GenAI and LLMs as your way of getting access to the patterns in the world, what's going to happen. The world will be a different place as that happens. I don't know if it's a good place or a bad place, but it's going to be a different place.

How do you learn the critical thinking? You need to be exposed to those things and you can't keep asking a machine and then you get the answer and then you get the satisfaction, oh, I got the answer. That's it. I'm done. How do you know that you can do more? And actually, how do you break down a problem? How do you develop that system thinking? We cannot let it happen that way because then what are we training? Next generation of brainwashed kids?

Having opened up about the fact that you've worked for universities, you think about the fundamental change they have to do now, because again, I only spent 6 months here. I spent very little time, but my view of those education facilities is they are running legacy education sessions. They are about giving you some knowledge and testing whether you got the knowledge, not giving you some tools and testing whether you can systematically apply those tools to solve a problem. And so I think our education facilities have to change.

Absolutely. And there are reasons for happening in the way that you describe, and one of the reasons being How do you mark if you test them on critical thinking at all times? You need massive power of assistants to, to be able to mark those exams. It was always a challenge. I'll give you an example for the advanced database course where I was teaching them query optimization and all those techniques. So to test them, I would give them an SQL query. And ask different query plans, different evaluations. And I would always provide them with the calculations. I don't care for them to memorize those things. And I just wanted to test how they apply that knowledge and how they go about it and do rewrites and whatnot. It, it's always very challenging to mark those kind of questions because they're open-ended. How do you compare this solution with that solution when they're both good solutions? And now we can tie it back to non-deterministic.

Well, maybe we tie it back to the pattern of education, which is based on a test that was graded to see whether you'd learnt is actually a pattern that no longer survives.

Exactly.

And what is the pattern? Is it actually they can tell you a story, they would solve a problem, and that actually is how you grade them, or do you need to grade people anymore? Again, all the patterns that are foundational for the last wee while are probably about to change. And so maybe the answer is more people write good books that tell better stories, and that becomes the content that all the universities actually use rather than the textbooks of old. I don't know.

And how do we teach them to learn to be better storytellers? Because that's art. It's an art form and you need to practice it. These kids, they don't read that many books anymore. They are creative in their own ways. We cannot expect them to behave in the way that we did because especially me growing up, a kid in communist Romania would get 10 minutes of cartoons Saturday at lunch and we had to run back from school. I was going to school on Saturdays too, so I would run back from school just to catch that 10 minutes of Bambi. And throughout half a year, I would see the whole movie. Books were the salvation. They grow differently and they're exposed to different things and they manifest that creativity in different ways. And it's all amazing as long as it keeps manifesting and they bring that into the world. That is a good thing. I completely agree with you. It's gonna be a big shift. And I think we are not the only ones talking about this, but we just get more vocal. About it. It's something that is on many people's minds. They are the future, so we need to prepare them for that future. And we are not sure exactly of that pattern of the future, so we have to prepare them for something that we guess how it's gonna be, but who knows.

One of the patterns I teach a lot of teams is skills, not roles. When we look at data teams, they often want roles, and data modeling is my favorite one to be snarky mc-snarky on, right? That's the data modeler, but why can't everybody model. Yeah, they can't be experts at it, but they can be proficient. So if we teach everybody data modeling skills and we no longer have a data modeling role, then the team's a better team. I think that's where we're going to get to, is that the education system has to teach skills, not roles. So actually, your example of going in and having to make a choice about going through arts or science, it's going to disappear. You're going to have some storytelling skills and some science system thinking skills, and that's going to allow you to craft yourself to fill one or many roles. And the ability to put those skills together as a set of patterns to solve certain complex problems is actually where I think we'll end up. But who knows? We did it with our parents where we were like, we're not going to work the way you did. The world's going to change and every generation is going to do that. Exciting time.

Yes. Exciting. We're not even thinking 5 years down the road right now. We are shortening that period of the forecasting. We just don't know, but it is exciting to live and start from scratch and learn these new things just at the same time as the new grad.

As I said in my LinkedIn post, when I announced that the book was finally live, I now know far more about Adobe InDesign bleeds and Amazon Kindle. Direct publishing bleed formats than I've ever wanted to know and ever really needed to know. Probably in hindsight, one of the skills I should have given to somebody else who had those skills rather than trying to work it out myself. Just to close it out, I suppose, I had a niche, I finally got around to fulfilling it, and I learnt so many new things in doing it because I had so many new problems to solve. That was intriguing for me. That was part of the journey.

So, Shane, if you were to summarize in one sentence, what advice would you give yourself at the beginning of the process, now that you are on the other side of it?

That's a hard one. I'd naturally go back to, now I know some of the patterns, don't do them the wrong way, don't do the anti-patterns. But actually, that wouldn't have been, it's actually the journey of—

That brought you to where, to the end of it, yes? So you had to go through that process. It was a transformative process and the flywheel and everything. So it would have been a different book if you didn't go through those iterations, don't you think?

It might've been the same book, but actually it would be a different learning experience for me.

Okay.

I got told right at the beginning, writing books are hard. Most people don't finish. And now I understand why. I got told at the beginning, don't write a book because you want to make money because you won't. And that's true. Don't write a book because you want to be famous because actually nobody's going to find it. Very rarely do you get the books that we all read and aspire to where it becomes effectively viral. There are so many books out there that probably like my one will get published and very few people will read it. So if you're going to go and write a book, do it for yourself because that's the one thing that'll be sticky is that it's something you want to do. And if you happen to make money out of it, great. If you happen to become that thought leader book that everybody reads and loves and you get massive positive feedback, great. There are good benefits that you can achieve, maybe. But if it's not something you really want to do, then you're not going to finish it.

I love that. Write the book for yourself. I love that.

Yeah. Cause once you've got a printed copy and it's sitting on your desk and you pick it up in 6 months' time. It should be something you're happy with because it's a lot of blood, sweat, and tears to get those bloody words on a page.

I wrote a book when I was very young and there are two copies, mine and the co-author. And that is because the publisher screwed us over big time.

I got to say that I've self-published. That was hard, but I think it's the right choice. Hearing some of the horror stories about publishers. And again, that was one of the learnings I had was I somehow thought if you used a publisher, they did a whole lot of heavy lifting for you, but all the research I've done, it's not true. They don't go and sell it for you. You have to sell it yourself. Yes, you get an editor and if you get a good editor, there is massive value in that. I think that's something that people have told me, but yeah, all the things you think a publisher does, they actually don't do. You still have to do it. I think there's value in publishers. I think the publishers would have given me a forcing function. They would have forced me to create the content at a certain cadence and that may have been valuable. Yeah, I'm quite glad I self-published. So we'll see.

Once again, congrats on scratching that itch, learning so much in the process, and thank you for sharing with everyone what you've learned and the process. I hope this will help everyone aspiring to write a book and to learn more about this.

Oh yeah, definitely. If people are struggling to capture data requirements, then Jill Data Guide to the Information Product Canvas is the best book in the world right now for understanding how to capture data requirements.

Excellent.

There's the last sales pitch. Thanks for taking over as the host. If people wanted to catch up with you and talk to you, where's the best place for them to find you?

The best place and probably the only place is on LinkedIn. That's where I spend the time. And yes, I'm on some Discords, but officially LinkedIn. Excellent.

I'll put your LinkedIn link in the show notes and thanks for coming on and taking the seat of host. And I hope everybody has It was a simply magical day.

Thank you, Shane. This was a lot of fun.

It was a lot of fun. Excellent.

Catch you later.

That, Data Magicians, was another Agile Data Podcast episode. If you want to learn new patterns that you can apply to craft your own agile data way of working, over to agiledata.io.