The pattern of writing a data book with Shane Gibson and Ramona C Truta - Episode #64

Foreign.

Welcome to the Agile Data Podcast where we talk about the merging of Agile Pro product and data patterns in a simply magical way. My first book, An Agile Data Guide to Information Product Canvas, is published on Amazon. If you hear your stakeholders say, hey, that's not what I wanted, and then you hear your team reply with, if that's what you asked for, then you probably have a problem capturing data requirements. The Information Product Canvas is a pattern template that is a quick and simple way to solve that problem. Head over to agiledataguides.com and click on the Green Book link to buy a copy now. Welcome to the Agile Data Podcast. I'm Shane Gibson.

And I'm Ramona C. Truta.

Hey Ramona. Thank you for coming on the podcast today. Today we have something slightly different because today I'm going to be the guest and you're going to be the host. So this is going to be a really interesting experience for me. But before we do that, why don't you give the audience a bit of background about yourself.

Okay. I have started in data since ever I joke that I came on this earth doing data and I'm going to explain. I've always been the kid who arranged all the fundraising, did all the financials. I was the trusted kid who really did every year, all of that. And so it was data, right? Not Excel spreadsheets, but with pen and paper. And I loved math and always dreamed of becoming a math teacher. At that time in Romania, we had to pass an admittance exam and I went to fill in the forms and they had three separate options. One was computer science, one was math, and the other one was applied math. And the idea is that everybody would write the same exam and based on the mark that you get, you will be distributed in your section. So as I was filling in that form, a friend of mine told me, why don't you put computer science first? Because you can do with computer science everything that you can do with math. And some, of course, I wrote that first. I got a very high mark. I got into the computer science program clueless as to what that meant. My first reaction to the first computer science class was this. X equals X plus one. Now, for a mathematician like me, I was dead in the water. I just. Let's just say I had a very terrible first computer science teacher and I struggled. I struggled with the computer science for a while and luckily for me, I excelled at math. So that really counterbalanced and all the logic and all those other courses. Long story short, I get to the databases course and coincidentally at the same time with the probability and statistics. So that saved me. Fast forward. My first job out of university was working with Data and FoxPro. I worked for high school and I was preparing everything from design to implementation. The next job for multinational, it was with progress, another database job. And my first project that I was working on was doing black box testing. So this application that we're developing was for a Dutch company and everything was in Dutch. I had to use two separate dictionaries because there was no Romanian to Dutch translation. I had to use English as an intermediary. I loved that job and they advanced me very quickly, put me in charge for the project. And that's the time when I got to a job in academia, which was actually my dream. And funny tidbit, I had to do a lecture, part of the series of interviews and my lecture was in data modeling. I always remember this. And I got to teach that. And then I met my future reason to move to Canada. I got a scholarship to come and do a PhD program at university of Toronto. And during that time I was in the database group. So I continued that. And my research was in XML XQuery. So we did that kind of research at that time. And I started teaching the database courses at the computer science department at University of Toronto. And I've got to teach other courses, software engineering and other courses to make up the schedule. I've been doing that for a while and then I stopped teaching and moved to a different university to do research. And I was a data person and that is where I really got to practice all the things that I've been teaching and more. And to this day, I guess that was my absolute favorite job. Fast forward to today. What I am now is a facilitator of knowledge. This is what I'm calling myself these days. I just love to do that in the sense that I take the tech talk like really what the technical people talk and translate it into business talk and vice versa. And. And I talk a lot of strategy, especially in the data and AI era that we live. There's a lot of translation that is needed to be done. You probably have witnessed this yourself. A lot of people talk like they know everything, but it's just what I call the foam on the top of the cappuccino. And I'm the person that goes down to the caffeine. So I removed the fluff, I removed the foam and I speak caffeine, like really strong espresso caffeine.

As a coffee lover, I'm there with You. So that idea of being a translator, I think we see people fall into that role over time. And it's typically people we've talked about our ages before we started recording, but people of our generation, our age because we spent years observing and learning and training our brains to recognize patterns and be able to go, oh, actually I can describe that quite clearly because I have all these frameworks and patterns that I can anchor to. I have these examples and I can bring those in so that I can explain something that's complex in a relatively simple way to somebody else who's never dealt with that complexity. And I think that is a really important role. Especially now with all the AI stuff coming, that ability to be able to provide that role is going to become far more valuable than it's ever been.

Absolutely, Shane, and I'm really glad that you brought up our conversation prior to recording because what we said, we've lived several decades and we've collected a lot of data points and data. We have these massive data sets and now we can build those knowledge graphs of everything that we have accumulated during this time. And while the new technologies may look new, but they are built on similar principles and we have a system thinking and then we just apply all those frameworks. And I think this is where we are really valuable because we have all this body of knowledge that we've accumulated and we've built and we shaped our minds.

Hopefully it's true because I'm really bad with a hammer and a screwdriver or any of those kind of. So I'm going to have to rely on the last decade of my working life still being knowledge, not the hard work. Okay, like I said, we're going to flip the script. You're going to be the host, I'm going to be the guest.

So first of all, Shane, huge congrats and I applaud you for finishing up the book and putting it out there. And I hope the book does well.

Lots of people have asked me why did I decide to write a book. And it was the scratching itch. It was just something that I decided I wanted to do. And so, yes, there's lots of knowledge in there. I hope people adopt. I hope it's successful. Somebody told me really early on you don't make money out of a book. And looking at how much you get paid, you don't. And so for me it was about scratching an itch. And the other thing people said was it's incredibly hard. Like lots of people start, never finish. And it's been a few years. And now I understand why. It's really interesting. You don't really understand how hard it is. It doesn't seem that hard, like to just write a book doesn't seem that.

Daunting at the beginning, especially with ChatGPT. Right. You just prompt it there and then the book writes itself, right? Yes.

Yeah.

And we know that there are books like that. Tell us a little bit about, first of all, the title and how you decided on this particular book. I know it's all about patterns. And I shared with you earlier that ever since I've met you and I've known you, it was all pattern this, pattern that pattern the other. And come on, what? Why does everything has to be a pattern with you? And it took me a while to understand what was a pattern to you. So take us to this journey, if you will. Yeah.

So let's start off with patterns. Right. There's a book that was written many years ago, a pattern language, and it was about patterns for building houses. And the concept of the pattern really is there's a common problem and there's a common solution that fixes that problem most of the time. And therefore, if you can articulate what that solution is to that problem, it becomes a pattern. If we think about a house, when we want to build a house, there's inherent set of patterns in that house. If I want to build a lounge, I am typically going to have a room that is a certain minimum size. It won't be a cupboard, it'll be bigger than that. It will typically have, depending where I live, lots of windows and light. It'll have a place for couches and seats. It probably, in the modern world, we have a place for a television. It may have a heating source, like a fire. If I compare the design of that room to a bathroom, a bathroom is typically going to have one window. And that window is probably going to be opaque because you don't want people peering in. If I live in a sunny place, I'm probably going to have a porch. I'm going to have an open space outside where I can sit, and it has some form of shelter from the sun, but it's open, or it has a series of windows I can open. Each one of those is a pattern. It's a solution to where I want to spend most of my time during the day, where I want to go clean myself, where I want to sit when it's sunny and fresh. And that's what the book was all about, was describing these patterns for architecture. And if we think about that one more step is if we were going to build a house, nine times out of ten we'd do architectural drawings, we'd do a blueprint, we would sketch some details that make us think about the layout and the patterns we're going to apply and also give us something to follow and iterate as we learn some more things. And so for me that was really intriguing. It was intriguing because a few decades ago I had a consulting company, so the usual five to 20 people, data and analysts, consulting company. And I used to get really frustrated that whenever we went into a new customer, we seem to be reinventing the wheel. We're like, oh, let's start from scratch. And I saw a massive amount of waste and inefficiency when we did that. And so my natural reaction was to go to methodologies. Yeah, oh, we'll write a methodology, we'll write this book of rules and everybody would follow the rules. It was a frickin epic failure. Nobody followed the rules. The rules didn't work half the time because every context was slightly different. The amount of money we spent trying to write the methodology was waste compared to how much we got out of it. And so I got really frustrated and then I lucked into this thing called Agile. Completely changes the way you think. Aligned with that came this idea of patterns of small chunks of proven solutions for a common problem and then what we call anti patterns, the times that we know that pattern actually introduces more waste than the problems it solved. So when I first started to write, I thought about what I wanted to write because I had many goes at starting that failed. I sat back and went, oh, I'll just write this big book of patterns for data. And that's actually the first thing I tried to do. It was an epic failure in terms of the outcome, but it was brilliant in terms of the learning that helped me finally finish a book.

Oh, okay. Why do you think that happened? What exactly made that failure happen? Or what have you learned from that aside from shrinking the knowledge and trying to fit something? Did it change your perspective or your approach? What influenced the most and what made you change what you took with you and you carry with you in the future?

So I think for me it was understanding the way that I'm driven. Like I said, I had a whole bunch of false starts many years ago, over many years it was like, okay, I want to write a book, it's a niche. And then I start to do something and nothing would happen. And one of the things everybody tells you to do is write the Outline of the book, you know, write the table of contents. And for some people that is a good approach. For me it was an epic failure because what I keep doing was I keep rewriting the table of contents. I'd write the table of contents, I go, yep. And I'd write two paragraphs of actual content. And I go, oh, that doesn't fit the structure of the book. So I'd go and seriously waste more time restructuring the book. And after a while I realized that was an anti pattern for me that create the table of contents. The other problem I had was I would write when I felt like it and then I would not write. And that consistency was not a good thing. So I decided that what I needed to do was a forcing function that would make me write consistently and not worry about the content yet. And so when I do forcing functions, what works for me is make them public. So I said, I'm going to write 60,000 words in six months. So that's around about 10,000 words a month. I am going to, on LinkedIn, publish my word count as a recurring theme so that I could hold this metric to myself of how many words I wrote. I am not going to care about the quality of the words. I'm not going to care about what the words are. I am just going to prove that in six months I could write 60,000 words. And I did that and I used substack and I created the Substack site called the Agile Data Book of Ways of Working. And that was just 60,000 words. Now what was interesting about that is it forced me to research. So I would write about a pattern for say, team design. And I'd go, yeah, okay, this is the way I design a team. Or actually the example I used is because I'd go, I can't describe it. It's one of those things, you go, what's agile? When you feel like it? And that's not a great piece of content. So I started doing research and then I found this research around fixed mindset and growth mindset and a book by Carol Dyke all around that. And what I could do by doing that research and doing that reading is I could see the patterns that were anchoring me and where they might have come from because I can't remember where I read them. I read lots of books, I watched lots of things, I talked to lots of people. I experimented, but I couldn't see the genesis of where the idea came from. And that actually made my content better, in my view, because now I could think about it. More I could rewrite and write some stuff. And for me, writing those 60,000 words was six months. It gave me a whole lot of process that I've used from then on, but also gave me a whole lot of content ideas and a whole lot of research. So that was the first step. 60,000 words of absolute crap to prove I could write 60,000 words.

And I think that's when we first met, right around that time. And I remember I was supposed to contribute to some sub stack and you said, oh, this pattern. And you gave it a name. And I read the thing and I said, oh, I know this thing. I just didn't know that it has a name. And I think in the perfectionist in me was upset that how come there is a thing that has a name and I do the thing, but I don't know the name. So this is where my frustration with everything, pattern and name came to some context. Because you know that perfectionist, why don't you know this thing? And I started beating myself up anyways. That's an aside, but it sounds to me like even in the first iteration where you were trying to come up with that table of contents, it really sounds to me like you are trying to figure out a pattern in the knowledge that you wanted. It's meta. The way I see it, that you try to approach everything from a pattern point of view, including the thing that describes the pattern. I don't know. In my mind it looks like that.

So I think now, in hindsight, after a couple of years, yeah, I think that's what I was doing, but I didn't know it at the time. And actually again, so forcing functions are really important for me, otherwise I just won't get stuff done. I was the kid at school that was finishing, as you say, at 2am in the morning because he hadn't done it for the last four weeks. That really hasn't changed in. One of the things that was intriguing me was why is it so hard to write a book? And one of the things I've got to now is actually the first book's hard, but the second book's harder. And so I've got a challenge on Can I write a second book in six months using the patterns that I've created for this book? So we'll see how that one goes. I don't think linearly, I don't think in a structured way. I think in chaos and randomness. And that's just the way I've always worked. And I am constantly distracted. So I had to find A writing technique that worked for that. And then I had to learn some new skills because I lasted six months at university when I was a kid and then I left and went to work. So I'd never really learned the research skills that you get from a good university. So what I found was I could write, but I couldn't justify what I was writing. And therefore I had to learn this whole new pattern of research. Whereas for you, I think you understand research intrinsically because you've done it all your life and so you had that foundational piece that I didn't have. I think the second thing is over the last couple of years, I tend to prefer to work with other people. I find it more invigorating, I find it more fun. I find having somebody else that I'm working with gives me those deadlines to force me to do that work. It's a forcing function. And I have tried multiple times to write this book with many different people and every one of them was an abstract failure. And it's really interesting. It's like, why is it me? Is it the book? Actually collaborating on a book is actually harder than writing it by yourself, I think. So I've got to resign myself now that every other book I'm going to write is just going to be me. But having said that, in the book that I did publish, there were a whole lot of example canvases that people generously spent their time creating without me. They've gone with the book. So actually the book is a collaboration. It just wasn't co written in the typical way you would co write a book. Which again is an intriguing pattern.

I was just gonna comment that that eventually you found your pattern to write this book. Something that worked for you and the collaborators. And we have a few friends who wrote books and probably they are agreeing with you on the side of having co authors and collaborators. And it's hard in general when you work with somebody and having different styles. And it's hard not for everyone. And it's a different set of skills that is required and always having to push towards your own issues. The other person issues.

I think the other thing is I'm highly opinionated and I had a certain vision for the book, like the layout. The book is about the information product canvas and that is an iteration of the business canvas. And I love the business canvas book when it came out, just the style, the form and the function of it. So this book is an iteration of that style. It is more coffee table than it is textbook. And I think if I ended up with a true co author, there would have been a hell of a lot of tension because I don't think I'd be willing to give up the structure of that book. And in fact, the early feedback I got on the structure of that book was that it was crap. And it's gonna be really interesting now it's published to see whether that's still true. But it's the book I wanted to write and like I said, I wrote it to scratch an itch.

Did you succeed? Hard scratch.

Yeah. Yeah. So definitely a hard scratch. Yeah. I liked the book more than I thought I would, which for me is a big compliment because normally I'm doing something and I hate it.

Okay, so, Diane, it's a success. You achieved what you set out to do. Yes. And you were happy with the result.

Yeah, but a couple years late.

But let's go back a little bit in the process and see what other patterns we are identifying, you say several years later. But there was a purpose or since you studied that fixed versus growth mindset, you had to transition towards more growth mindset than the fixed one. So that gave you plenty of opportunity. So just consider how much growth there is and how much you got out of that process on its own to achieve that kind of shift, because that is huge. And especially as we grow older, if we are a fixed mindset, then the older we are, the older we get, the much harder it is to shift to the other side. So I wanna congratulate you on that and give yourself grace for it. It's a huge achievement.

And I think the fact that actually I finished it, because I do tend to start some things and not finish them on a regular basis.

Shane, let's admit that's a pattern that I don't know anyone who doesn't follow it, let's put it that way. And especially now with AI so easy to experiment and oh, I have a weekend. Let me VOD code something. I know your op. I did about that. I read what you wrote. Because everything is so easy and then you start something and then the mind goes to the next thing and the next thing. And at the end of the weekend, okay, what do I have to show for. And this is something that I bring up more and want to talk about the new paradigm or pattern, to use your preferred term, of learning how to learn. There's a lot of shift in how we learn now in this context. And for us old timers, we have a lot of knowledge as we established at the beginning and now with the VIBE coding And all these tools that do a lot for us, we immediately can make connections. Oh, it's doing this even though it's something that we never saw before. Code. We never want to learn and do it ourselves. But we have anchors and there are things that we have knowledge about. We may not know syntax and stuff, but we understand what's happening. And then we can go and ask in a chatbot, hey, tell me more about this and teach me more about this, and then check my knowledge. I do that for myself. It's a lot of fun. And I don't know that I've ever been this about learning new things as I am now.

We've got access to a knowledge base in a way we've never had before. I'm old enough to remember the Encyclopaedia Britannica as the thing you used to have to go and look up. You wanted to understand what a thing was. You know, you used to have to go to the library at school and get a book out and read it and everything's changed. I think there's some interesting concepts out there now that with the advent of the LLMs and the Gen AI and the Access to that knowledge that curators of that knowledge, humans that actually tell a story based on that knowledge in a different way will actually become the more successful people. Having the knowledge is no longer good enough because that knowledge is now available at a click of a button, type of a key. It's the way you tell the story that becomes intriguing. So we're going back to thousands of years ago to storytellers being the most valuable part of the tribe. And that's intriguing. So time will tell.

Intriguing how?

Because it goes back to an old pattern. It goes back to the storytellers and the way they told the story becoming more important than the holding of the knowledge. The articulation, the ability to have somebody else understand what you understand, to coach and mentor and teach, because the knowledge is available to anyone.

And it's just the way we transfer that knowledge. Going back to what I was saying at the beginning, that I'm a facilitator of knowledge and I'm a visual thinker and a systems thinker and all sorts of other things. And what I observed a pattern. I observed this pattern that I have conversation with many people, just as you said, right? You talk to a lot of people. I read a lot, I see a lot, I consume a lot of knowledge. And suddenly it's all that chaos gets into an equilibrium and I get a visual, a simple sentence. But I get a picture, and that picture can be described in so many words, and I can use it. It's a pattern or a framework on its own that I keep using, and I keep finding usage for it. Like the cappuccino thing or another one I came with was the puzzle versus Lego. In this highly siloed world we live, everybody's working on their own tiny piece of puzzle, and then they expect that they all get glued together somehow, or everything fits and expect to work like Lego, which, no matter which Lego you buy, which LEGO set, they will fit on top of another. And you can build something, but that's not the reality. You know how Joe encourages everyone to write an article explaining why they write or their process? So I was 13, and in Romania, we had to give an exam at the end of grade 8 to go to high school. So we had to decide what kind of high school we want to go, etc. And at 13, I remember vividly I had this conversation in the schoolyard with my Romanian literature teacher. And she asked me, so, Ramona, what are you gonna do, storytelling or math? I was exceptionally good at storytelling. I know how that comes across. But it was effortless for me. And I would go to competitions, Olympiads, I could never qualify for math to higher levels, but for storytelling, I would just go, and I would always win. So I had to make that decision. And what hold me back for going on to the humanities side of training was the fact that I didn't feel good enough for a foreign language. So my first foreign language that I had to get in school was Russian, then French. But I had such a terrible teacher that we were clueless with French. I've learned French going to high school, so I picked it up. English when I went to university, so I go into the sciences. I did math and physics. And I remember years later, in grade 11, I'm in this class with these kids. Everybody was anybody in my little town. And we have the Romanian literature exam, grade 11, and we are given a fragment of one of the books that we've been studying that trimester, and to write an essay. And when the teacher brings back the exams, she acknowledges me in front of the whole class. She highlights my work and my essays and my storytelling and all of that. And I thought, huh, how interesting, because I was never acknowledged in that way for that particular subject. And it was the only time she ever gave me the 10 out of 10 mark that stuck with me. And here I am, decades later, finding myself in that passion of telling stories again. And Want to tell stories about so many things.

I remember years ago I used to work for sas, the software company, statistical company. And I was in pre sales so I wasn't in the implementation side, but I spent a lot of time with back then they were called data miners. And one of the techniques they had that still sticks with me was they would go and build an analytical model using a neural net and they'd do it with a bunch of different stats techniques. Often it was marketing related. So they'd look at the lift chart and they'd go, that model with those parameters is the best fit for solving that business problem. And sometimes the neural net would be the one that wins. The problem was you could never explain to the customer what the hell it was doing because how does it work? Well, basically we push all this data in, we set these three flags, we let it run, it came back with the best answer. So what they used to do was they used to produce a decision tree model at the same time and they used to use the decision tree to explain to the customer what's happening. And they would say this is not the model, but the model is doing something like this. It's looking at customer and it's looking at length of time they've been with us and then total spend and the number of contact center calls to come up with these clusters. And although it wasn't the actual model, it was able to tell the story that the model was probably doing. And so for me, that's always stuck with me is that we have to be able to take these technical things we do and explain them by just telling stories. And when you're writing a book, for me it became actually how do I tell a story? Because I know the techniques, the patterns that I was using because I've used them for a decade with lots of people I knew they worked. But there's this really weird thing that when you're forced to write, actually it helps enhance your understanding of what you know. And so on this podcast, what I do now for this year, I will say to somebody who's coming on as a guest write an article about the pattern we're going to talk about. Not because I need to see it, but that art of writing, that process of writing the article about what you think you know will make you know it so much better. And for me that was something that I didn't expect because I my whole life has been talking and waving hands and drawing pictures compared to math or writing and so moving into the skill that I didn't have. And I still don't think I have it very well, but it helped me articulate and understand what I thought. I knew a lot more because once you've written it and you go, that's just pants. I don't understand what I wrote. How does anybody else do it? I think that is actually part of the value of writing. So again, something new for me.

You've explained that really well because we have so much in here, right? So much as we've accumulated. And I see this in all the conversation, especially with tech people, and they immediately go to tech jargon. And what I want is I want to hear a story that is tech agnostic. Can you distill what you're saying without all the jargon? And I've been in meetings, sometimes we were in those meetings together, and at all times I had to figure out a translation in my head, oh, this person is talking about that and this person is talking about the other thing. But I think they talk about the same thing. But they can figure out that they're talking about the same thing because they use their own particular jargon. So that was a big shift for me, understanding that people around me are talking pretty much the same thing. They just don't use the same vocabulary or they're very particular and very opinion opinionated on their respective jargons. That's why I became so passionate. I've always been a teacher. I've taught thousands of students and had to explain and distill and all of that. But it gives me so much more passion right now, realizing how people just touch the surface of things and don't know what's underneath, but they would like to know. And they just need a different way of explaining that without all the jargon and all the technical details. You just want to understand what's in that box at a level that allows you to have a conversation with somebody else or a convincing conversation.

And one of the interesting things for me was, so I started off with 60,000 words and I looked and said, actually, if I documented every patent that I think I know, it'll be 600,000, 6 million, and it's never going to happen. I then picked one of those areas, which was the canvas, and I started writing the book. And it got too big again, because a pattern, especially in the data world, doesn't live on its own. So the canvas is a pattern, but actually there's a workshopping pattern. There's the who does what pattern, the core business event model patterns already in there, prioritization Personas. There were all these other patterns that it used, it leveraged. And so I ended up again with a book that was too big and I had to stop myself and refocus. And then the other thing was, whenever I used a term, I started to go into too much detail. For me, it was like, oh, I'm going to go and explain it to the nth degree. And again, that made the book too big. I was going, if I keep writing at that level, I'm never going to finish. And so what I decided was to write a book for people that were inquisitive. So it tells you what you need to know, but actually you still have to go and do more work. And it's really interesting. As part of our startup, we have people that are in the middle of their degrees from around the world come and do work with us for three months. And I remember I had one student and we give them a bunch of choices of what they can work on. And he said he wants to get into product analytics. And I said, okay, you've got two choices. You can do product analytics or marketing analytics. For a product, you need to pick which one that you're going to do. And he goes, what's the difference? And I went, actually your first task is to go and give me back one page that describes the difference between product analytics and marketing analytics. And I waited and I waited and a week later I went, where is it? And he goes, I couldn't find anything. And 10 seconds later, being grumpy old man that I am, I googled it, I found 20 references and I sent it back to them them. But what that meant was he wasn't interested, he wasn't inquisitive. He was only doing this to pass a tick in the box. He didn't actually want to learn. And so I think again, that's part of it, is that books are there to help you learn the next thing you need to learn. But that second step, going from the 60,000 words to what I thought was very focused and then starting to write and then realized, no, it's still too big, forced me to go to that third iteration which was, what am I not going to write? And that was more important. The constraints based model of I am not going to write about these things was the last thing that I had to understand in my writing process. So start really big, lots of stickies, complete chaos. Find something that looks like it coalesced into a great story that I could tell that was different to every other story that I'd read. Start to write that story and realize it's still too big, there's too many 10 rules, the graph's got too many nodes, and then a constraints based model to say what am I not going to do? And then that was the process that let me get to 193 pages in a couple of years. And the test like you said, is I think I have a pattern to write the next book. I know what the subject is. Can I reuse that writing process that I think I have to do? 193 pages in six months. That's going to be the test for me because that's when these are pattern, not just a bunch of ad hocs.

So Shane, during the process of writing this book, have you uncovered some new pattern that you didn't know before, something that really excited you so much that you wanted to include and maybe something that is one of your preferred patterns right now?

Yeah, there's a couple. So one of the things that happened when I was writing the book was I had created a course for the canvas which I used to teach privately to organizations and I made public on Maven. And what I didn't realize was running that course helped me refine the content and it also helped me get feedback on that content because effectively I was practicing delivering that content time and time again. So what I'm going to do next time is I'm going to write the course first and then write the book. And the second thing is I'm going to focus a lot more on the questions that have been asked via the course because I didn't record them this time. And what I found was actually the questions that I get asked is the content the book should focus on. So that's the first thing in terms of the writing process. Create that course, run it, iterate the content on the course. As I'm entering the content of the book, actually document or record the questions I get asked by the students because that's telling me what I'm not articulating. The second part was the example. Canvases in the book have been done by a whole lot of collaborators based on projects they've done. And I hadn't written the book so I had nothing to give them and they hadn't been on the course. So what I did was I took them through the canvas as a quick half hour. Here's how you fill it out and then let them go off on their own and do their own canvas. And that was intriguing because they did it in ways I didn't expect. But there were Also patterns across every collaborator. So, for example, in the canvas, there's a vision statement, and the last part of the vision statement is what's called unlike. So if we didn't build this information product, if we didn't deliver this data and information to the organization, what would happen? What's the alternative negative thing that would happen? And pretty much for every collaborator who worked on their own, the answer was manual process using Excel.

You're kidding me.

No. Because every project they worked on, because they're all practitioners in the data domain, that was what they were replacing. And it made me think. We talk about the value of data and information, and we've talked about that a lot lately. We talk about the value of data teams, we talk about writing business cases and justification for this work to be done. But actually maybe we're overthinking it, because really the alternative is, do you want to run that process in Excel or do you not? Because that's actually the thing we're replacing nine times out of 10. And that was intriguing for me, that spawned a whole lot of thinking about value of data in an organization. Are we replacing a process that's inefficient and do people really care? So that was another kind of surprise learning for me as part of that process.

So really, what do you do with that? So which answer did you settle on if you ask these questions? Right. So are we over exaggerating things or are we over emphasizing on the value that we bring? How do you settle seeing this pattern emerging from your collaborators? And those are the people at the ground level, right? Ground zero. They are doing this. How did you settle that pattern emerging?

I haven't answered it. It started me thinking. It started me thinking. Data teams are like a finance team. Data teams are like a finance team without the mandate. A finance team, a bunch financial people in the financial part of the organization are a shared service and they're mandated. You can't typically run an organization without a chief financial officer, without people doing the accounts in the general ledger and producing profit and loss and balance sheet statements. They don't articulate their value to the organization. They don't go, would you like to have a balance sheet and a profit and loss? But for some reason, data people do. We're a shared service and then we have to go and justify our existence. And that seems like an oxymoron to me. It's intriguing. Maybe the answer is actually we do get embedded with the operational team. We no longer become a centralized shared service. We become decentralized. I Don't know. There's lots of questions there. And when we do Data Team Design book, the guide for that'll probably. I'll have to answer them at that stage. That's not the next book, by the way. That one's a hard one. Yeah. Again, the good thing about it is that as you write, you learn because you're thinking, you're asking questions, and then you're researching and exploring and collaborating. And so just the process itself is invigorating.

This was a really perplexing pattern to emerge. Right. I know we always joke that, oh, Excel will never die and Excel is all you need. Right. So not attention and Transformers. Excel is all you need. And to actually have this end result of all this research and the work that you and your collaborators have done, you validated that which we took as not empirical, but we as anecdotal information. So you actually proved that is a fact. It's steeped in reality. It's really fascinating.

And these are people that are professionals and typically consultants that get paid to do that work. An organization has paid them or their teams or their companies to replace a manual Excel spreadsheet process with something that's quite costly. Because data teams are costly. Intriguing.

Yeah, very. And so in this context, Shane, you just proved that data teams are a cost center, essentially. So, yeah, it's. It's not the conclusion that you would have hoped to arrive at, but now we have AI and we need data teams to put the data in order so we can consume all that money to generate AI to fix something that we can do in manual Excel. Is that what you're saying?

I'm not convinced that AI, the gen AI, LLMs, and the new way of working with those tools actually equates to us structuring our data more. I have a hypothesis that I haven't got around to testing that actually will structure our data less. But that's probably a different and very longer podcast.

Okay, so maybe we're looking at it from different angles. Let's clarify this a little bit, because structuring or cleaning the data, those are different aspects of it. So for any data to be relevant and to get patterns, good patterns out of it so we can learn something from it, it will have to be better than just plain garbage.

I'm not sure it does.

Please elaborate.

So this is unstructured thinking. I haven't gone through and done the work, the research, the experimentation to get core patterns out. So I'm very early in my thought process. But the LLMs have been trained on pattern recognition of large text repositories. That's what they do. So why do we think that our data factories, our way of working, where we put all these processes in to change the structure and the context and the content of data, increase the quality, is actually a better way of feeding those models than just giving it the chaos and asking it to find the pattern? That's my underlying hypothesis and I don't know the answer to it.

As you said, this should be a podcast on its own. And Mr. Snarky McSnarky came out to play. Oh yeah, I was wondering when he will come out to play.

I think we have a whole new set of tools that have a whole new set of patterns and there's a real risk that we're trying to apply legacy patents that haven't worked for us anyway into this new world. And I just think that's dangerous. So I don't know. Again, we'll find out in the next year or two because things accelerating so.

Fast, we are talking about data in general and how it's shaped and all of that, but in the context of a business, that data is pretty structured, or we want to believe that it is, right. And all the documentation, everything that, that we can think it's digitized already or about to be digitized, it is in some shape and form better than how we imagine everything that was scraped off the Internet. It's a different kind of data volume sets, whatnot, when we talk in the context of the business and the information that is pertain inside the business and the rest of it.

So let me give you a concrete example that keeps coming back to my head. Again, I don't come from the library services domain and so whenever I hear ontology and taxonomy, I get a whole bunch of technical words that I don't quite understand. That's not true. I kind of understand them more now because I had one on one of the podcasts to explain to me what the hell all those things mean. But they're about putting a form of structure against some things, a classification to a degree. And my question is that a better pattern for using that type of data with Genai and LLMs versus just chaos tagging? If I went in and just tagged the snot out of every piece of data that had any kind of value with any tag that it might be, and then put that chaos into the LLM, will I get a better result? And that's my question, is that actually there's more unstructured chaos to let it find the patterns Is that better? For example, it's the difference between a K means where I just throw all the data at it and say tell me four clusters versus a SQL statement where I put a Nelson to say here's four clusters. I'm just intrigued on the difference of those patterns right now. And I think we are applying the tools that we know to a problem that we haven't proven. That's my view at the moment.

Well, I guess time will tell. It's the entire non deterministic nature of it it and there have been so many papers coming out and there are examples of one shot learning when it saw it once and that is what it learned as opposed to other things that have seen it many times and it didn't learn. So that is an issue because you don't know in the end what it will learn, what it will be the end result among the structure, a part of the equation. And this is just, just how my brain works. And that is what makes sense to me. But I understand and I know there is order and structure in chaos. You just have to find it. And that requires a different technique, different set of patterns.

Yeah. And Juan Cicada challenged that versus non deterministic model in a really interesting way. What he said was because we know the machine is not deterministic, because we know we will ask it a question and get a different answer. We don't trust it because we expect the machine to be 100% right. However, when we ask a human, when we ask a data analyst to analyze that data and give us the answer, we don't expect 100% accuracy. We don't ask to see their workings, we don't ask them to prove it. We trust that they know what to do with their job. So there's this dichotomy between not trusting a machine but trusting a human. Whereas actually a human is probably far more non deterministic than a machine to a degree. And an example today on the practical data modeling discord that I posted was I saw somebody post something on LinkedIn that said the technical tools that a data scientist use is Excel and tableau. Like bollocks. The term data scientist should be the clue that there's some steady science involved. And they actually listed, they listed the stat skills that a data scientist needed. So they said the skills you need is stats. And they listed a whole lot of stats, capability and patterns you needed. And then they said the tools is excellent tableau. And so I can look at that and I can go that's a disconnect that doesn't ring true to me. I don't believe that based on my experience. But how many other people would? And I think we seem to think that humans are infallible and we should trust ourselves where actually we can't. Yet we won't trust the machine because we know it hallucinates. I'm intrigued again by that dichotomy.

I don't know who created that. And there were some other issues there. If you went to the right most column with the gen AI engineer and then there were some other typos there, probably it was generated, who knows. But back to this idea of humans being more non deterministic and the machines being non deterministic. So why do we trust humans more than machines? I've heard Cassie talking about this. She wrote several articles on this idea that. But it's not that it's more non deterministic, but there are a lot more correct answers. To go back to that, which one do we choose better? Out of all these good answers, correct answers, which one is more correct? So they are all correct. So how do you choose? How do you make decisions? And especially from a leadership standpoint, what do you do with all of that? You have to make decisions, you have to. To entrust in the end in something and anchor and ground. What is the ground true? What do you do? And it's an interesting point. With humans, we have such a large spectrum. You put two different people with the same title next to each other and you will need a translator in between them.

We just got to take the arguments we have about what is a semantic layer, our semantic arguments on semantics.

You were in that meeting?

Yes, I think so. And then you got the what's a data product Argument.

And then in that meeting with the semantics that. Okay, what exactly are you talking about? From where I'm coming from, the academic side and the research, this is what I think of semantics. What are you talking about? And then it was complete silence. And I didn't know what to do with that silence.

But because you can speak multiple languages, you are doing a form of semantic mapping to be able to take Romanian and make it French or English or Russian. Your brain has been wired to that mapping. And one of the things that I look back on my life and go, I wish I learned a language at five years old, a second language, because I think fundamentally that would wire your brain. Yeah, maybe three. I don't know what the age is for the brain wiring, but to learn a second language fluently at a really young age, I think would wire your brain completely different and actually allow your ability to semantically map different patterns in the world to happen a lot faster than if you hadn't learned multiple languages. I'm intrigued by that one.

My daughter, she is so interested in languages. Well, languages and more into symbols and alphabets and scripts. So she had taught herself more than 10 scripts. She wanted to call herself a poly Alphabet, so she's now a poly Alphabet and she explains it scientifically. So she did all this research prior to ChatGPT on her own and she would tell me that sound and that phonetic and this and that and wow. So it's all very systematic. And she found the rules, patterns. I know I kept throwing patterns at you as an inside joke between us, but now I'm serious. She uncovered patterns and she could figure out connections between a pattern in some script and patterns in the other script and how they translate. I'm realizing this in real time in my conversation with you right now.

Oh, and she's multimodal, right? Because she's doing it with both text and audio. So she's got the ability to be a multimodal.

Yeah. And she really wants to stay away from AI. And Joe talks about his own kids and how, no, they insist on not being influenced by AI and what is this doing and that. And we on the other side, we embrace it and we see how much help we get. And that's another pattern emerging, I think.

Because we've had to do the hard work ourselves. Therefore I think we can leverage the tool. I wonder if you grow up with knowing nothing other than gen AI and LLMs as your way of getting access to the patterns in the world, what's going to happen? The world will be a different place as that happens. I don't know if it's a waste or a bad place, but it's going to be a different place.

How do you learn the critical thinking? You need to be exposed to those things and you can keep asking a machine and then you get the answer, and then you get the satisfaction, oh, I got the answer. That's it, I'm done. How do you know that you can do more? And actually, how do you break down a problem? How do you develop that system thinking we cannot let it happen that way because then what are we training next generation of brainwashed kids?

Having opened up about the fact that you've worked for universities, you think about the fundamental change they have to do now, because again, I only spent six months there. I spent very little time. But my view of those education facilities is they are running legacy education sessions. They are about giving you some knowledge and testing whether you got the knowledge, not giving you some tools and testing whether you can systematically apply those tools to solve a problem. And so I think our education facilities have to change.

Absolutely. And there are reasons for happening in the way that you describe. And one of the reasons being how do you mark. If you test them on critical thinking at all times, you need massive power of assistance to be able to mark those exams. It was always a challenge. I'll give you an example for the advanced database course where I was teaching them query optimization and all those techniques. So to test them I would give them an SQL query and ask different query plans, different evaluations and I would always provide them with the calculations. I don't care for them to memorize those things things. And I just wanted to test how they apply that knowledge and how they go about it and do rewrites and whatnot. It's always very challenging to mark those kind of questions because they're open ended. How do you compare this solution with that solution when they're both good solutions? And now we can tie it back to non deterministic.

Well, maybe we take it back to the pattern of education which is based on a test that was graded to see whether you'd learned is actually a pattern that no longer survives.

Exactly.

And what is the pattern? Is it actually they can tell you a story, they would solve a problem and that actually is how you grade them or do you need to grade people anymore? Again, all the patterns that are foundational for the last we while are probably about to change. And so maybe the answer is more people write good books that tell better stories and that becomes the content that all the universities actually use rather than the textbooks of old. I don't know.

And how do we teach them to learn to be better storytellers? Because that's art. It's an art form and you need to practice it. These kids, they don't read that many books anymore. They are creative in their own ways. We cannot expect them to behave in the way that we did because especially me growing up. A kid in communist Romania would get 10 minutes of cartoons Saturday at lunch and we had to run back from school. I was going to school on Saturdays too. So I would run back from school just to catch that 10 minutes of Bambi. And throughout half a year I would see the whole movie. Books were the salvation. They grow differently and they are exposed to different things and they manifest that creativity in different ways. And it's all amazing. As long as it keeps manifesting and they bring that into the world, that is a good thing. I completely agree with you. It's gonna be a big shift and I think we are not the only ones talking about this, but we just get more vocal about it. It's something that is on many people's minds. They are the future, so we need to prepare them for that future. And we're not sure exactly of that pattern of the future. So we have to prepare them for something that we guess how it's gonna be. But who knows?

One of the patterns I teach a lot of teams is skills, not roles. When we look at data teams, they often want roles and data modeling is my favorite one to be snarky mcsnarky on. Right. That's the data modeler. But why can't everybody model? Yeah, they can't be experts at it, but they can be proficient. So if we teach everybody data modeling skills and we no longer have a data modeling role, then the team's a better team. I think that's where we're going to get to is that the education system has to teach skills, not roles. So actually your example of going in and having make a choice about going through arts or science, it's going to disappear. You're going to have some storytelling skills and some science systems thinking skills, and that's going to allow you to craft yourself to fill one or many roles. And the ability to put those skills together as a set of patterns to solve certain complex problems is actually where I think we'll end up. But who knows? We did it with our parents where we were like, we're not going to work the way you did. The world's going to change and every generation is going to do that. Exciting time.

Yes, exciting. We're not even thinking five years down the road right now we are shortening that period of the forecasting we just don't know. But it is exciting to live and start from scratch and learn these new things just at the same time as the new grad.

As I said in my LinkedIn post when I announced that the book was finally live, I now know far More about Adobe InDesign bleeds and Amazon Kindle Direct Publishing Bleed Formats than I've ever wanted to know and ever really needed to know. Probably in hindsight one of the skills I should have given to somebody else who had those skills, rather than trying to work it out myself, just close it out, I suppose. I had niche. I finally got around to fulfilling it and I learned so many new things in doing it because I had so many new problems to solve. That was intriguing for me. That was part of the journey.

So Shane, if you were to summarize in one sentence, what advice would you give yourself at the beginning of the process now that you are on the other side of it?

It's a hard one. I'd naturally go back to now. I know some of the patterns, don't do them the wrong way, don't do the anti patterns. But actually that wouldn't have been. It's actually the journey of.

That brought you to where? To the end of it. Yes. So you had to go through that process. It was a transformative process and the flywheel and everything. So it would have been a different book if you didn't go through those iterations. Don't you think it might have been.

The same book but actually it would be a different learning experience.

Okay.

I got told right at the beginning writing books are hard. Most people don't finish and now I understand why. I got told at the beginning don't write a book because you want to make money, because you won't. And that's true. Don't write a book because you want to be famous because actually nobody's going to find it. Very rarely do you get the books that we all read and aspire to where it becomes effectively viral. There are so many books out there that probably like my one will get published and very few people will read it. So if you're going to go and write a book, do it for yourself because that's the one thing that'll be sticky is that it's something you want to do and if you happen to make money out of it, great. If you happen to become that thought leader book that everybody reads and loves and you get messages, positive feedback, great, they're good benefits that you can achieve maybe. But if it's not something you really want to do, then you're not going to finish it.

I love that. Write the book for yourself. I love that.

Yeah. Because once you've got a printed copy and it's sitting on your desk and you pick it up in six months time, it should be something you're happy with because it's a lot of blood, sweat and tears to get those bloody words on a page.

I wrote a book when I was very young and there are two copies, mine and the co author and that is because the publisher screwed us over big time.

I got to say that I've self published. That was hard but I think it's the right choice. Hearing some of the horror stories about publishers and again, that was one of the learnings I had was I somehow thought if you used a publisher, they did a whole lot of heavy lifting for you. But all the research I've done, it's not true. They don't go and sell it for you. You have to sell it yourself. Yes, you get an editor and if you get a good editor, there is massive value in that. I think that's something that people have told me. But yeah, all the things you think a publisher does, they actually don't do. You still have to do it. I think there's value in publishers. I think the publishers would have given me a forcing function. They would have forced me to create the content at a certain cadence and that may have been valuable. Yeah, I'm quite glad I self published, so we'll see.

Once again, congrats on scratching that itch learning so much in the process and thank you for sharing with everyone what you've learned and the process. I hope this will help everyone aspiring to write a book and to learn more about this.

Definitely. If people are struggling to capture data requirements, then Dual Data Guide to the Information Product Canvas is the best book in the world right now for understanding how to capture data requirements.

Excellent.

There's the last sales pitch. Thanks for taking over as the host. If people wanted to catch up with you and talk to you, where's the best place for them to find you?

The best place and probably the only place is on LinkedIn. That's where I spend the time. And yes, I'm on some discords, but officially LinkedIn.

Excellent. I'll put your LinkedIn link in the show notes. And thanks for coming on and taking the seat of host and I hope everybody has a simply magical day.

Thank you, Shane. This was a lot of fun.

It was a lot of fun. Excellent. Catch you later. And that Data Magicians was another Agile Data podcast episode. If you want to learn new patterns that you can apply to craft your own Agile Data way of working, head over to Agile Data Data IO.