#260 Harnessing the Power of Now With Real-Time Analytics with Zuzanna Stamirowska & Hélène Stanway

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com how fast can a human make a decision Right. For example, this is something that allows all the human operators and systems to be reactive to change, still collect all the data and actually process it and saturate it. It's not for reporting, it's actually for operational decisions being made.

Welcome to Data Framed. This is Ritchie. Real time analytics is one of the hottest ideas in data right now. The idea is pretty simple. If you don't have to wait for your analysis results, then you can do things faster. Of course, there's a big jump from asking a data analyst a question and waiting a week for them to run an analysis and getting results in milliseconds. I want to know how fast your analysis needs to be to count as real time, when you might want to go that fast, and how you need to change your analytics setup to achieve it. I have two guests with very different backgrounds for you. First we have Zuzanna Stamirovska, the CEO and co founder of Pathway, an AI streaming platform. She's building a company to help AI builders provide real time solutions. And Susannah founded the company straight from completing a PhD where she worked on forecasting maritime trademark. Next up is Elaine Stanway, an insurance industry consultant with HMLS Consulting. She advises insurance companies on real time, Internet of Things, and digital twin use cases. She's also president of the SENS Consortium for Insurance Industry Thought Leaders. And previously Helene was a director at edii, an insurance industry education company. So let's find out how to go faster. Hopefully the guests can answer my questions in real time. Hi there Helene and Susanna. Welcome to the show.

Great to be here.

Hello.

Just to begin with, I'm never quite sure how fast real time analytics is supposed to be. Does it mean instantaneous or is it something else? So can you just tell me what does real time analytics mean to you? Suzanne, do you want to go first?

So real time analytics can mean different things depending on the industry. So of course we can talk about it real time when we're talking about in a couple of milliseconds that are the very critical use cases. Like imagine in Formula one, you don't want your engine to just explode during the race, so you should be able to process all the data and actually get an alert that, okay, something is going wrong as soon as possible. So here we're talking about milliseconds. But very often people who really care about this level of latency, they would go as deep as the hardware to actually optimize for latency. And there are those cases. But then actually many more cases, what we see is, let's say sub second latency or even, you know, it may mean for some people it means second, but we're moving more towards just analytics being operational. So like, how fast can a human make a decision? Right. For example, this is something that allows all the human operators and systems to be reactive to change, still collect all the data and actually process it in such a way that it's not for reporting, it's actually for operational decisions being made. So let's think about real time and logistics. We're not at millisecond engines blowing up or level, but still we want to know as soon as possible, because as the data comes in, like some intervals, we want to know that, let's say a box of vaccines is about to get spoiled because it was just forgotten somewhere on the site. Right. So, I mean, generally speaking, I would say ideally we're like sub seconds to say really real time and milliseconds in the critical cases.

Okay, that's quite a broad range and I can certainly see how you don't want to wait several seconds if you're on a formula on track and then find out your engine's exploded. But yeah, certainly I like the idea of it just being faster than an analyst because very often with data science, you know, you ask someone a question, they come back with an answer a week later and that's very much not real time. Okay, so I'd love to talk a bit about use cases and who's making use of real time analytics. You gave a few examples there. Helene, can you just expand on this? Like, where have you seen real time analytics being used?

So to give context. So I work in the insurance industry, which is traditional in many, many aspects. But what is really, really fascinating for me is that the client base that we as an industry underwrite and insure and risk manage is moving at a faster pace than the insurance industry can deliver. Changes in insurance products and services to cater for that, that having been said, the client base is also knowing their risk much better than us as an industry. And it's from a use case perspective, we're seeing it across the board where there are fixed assets. I say fixed, it could be a ship that moves, right. It could be a building, it could be anything. That is the tangible asset and it's being used. And I really like Susannah's example earlier around the vaccines. So one such use case is how do you monitor the temperature and condition of vaccines as they go through from production right through into the arm? So where in the value chain was it failing from a temperature perspective? And two things from that. One is actually I need to send some more vaccines down the same pipe, down the same supply chain so that you can get individuals vaccinated. But actually is it always failing at one particular point? Do we need to go and change the freezer? I'm being simplistic in my example, but do we need to change the freezer? Do we need to do something else? So there's a whole bunch of examples. And actually again, just going back to the real time original question, as insurers, we probably don't need real time, real time because if a tap is dripping, it's probably going to drip for a little bit before we perhaps need to think about sending a plumber in for it to not be a flood. But there's so many use cases. And one of the ways that we drive and create use cases in the industry is actually looking at where there's been a claim. So where something bad has happened, could a sensor and therefore real time data have prevented or mitigated or made it not quite as bad as it ended up being? So there's a whole plethora of use cases, but the ones I'm most familiar with are in chips in commercial properties and also supply chain.

That's really interesting. And I have to say I don't generally think of the insurance industry as being really cutting edge in terms of analytics and how fast, like quantity can do things quickly. So it's interesting that even in insurance there are these very important use cases. So it sounds like supply chain is an incredibly big use case and that goes beyond insurance as like almost any company with physical products needs to worry about this stuff. So that's really interesting. Suzanne, have you seen any other examples of where real time analytics is important?

Absolutely. So when we're talking about the critical use cases, the financial sector supplies a wide variety of those. So everything that's linked to trading or pretty much your Competitive advantage is the ability to make a better decision faster. This is pretty much where you want to shave off latency. The classical is of course fraud detection and credit card transactions. You're there at the counter, you can play it all with the shop assistant and they're like shaving off, let's say 2 milliseconds in terms of latency is worth everything. Actually, I can't say much more about this, but pathways being used by NATO, you can imagine that there are many critical, let's say time sensitive decisions to be made in a larger context of operations. So these are some of the most critical ones. And then actually we do know pretty well everything that's linked to moving assets and physical assets. And it's true, like there's a number of things. There's the anomaly detection on the process itself where perhaps it's not milliseconds, but still latency matters because you know, this package of vaccines can be very, very precious. And indeed any sort of process analytics, they're being done to enhance them where you. And that's true. It's actually at say a certain scale of use cases and latencies that you would require that can also inform your technological choices.

And it seems like there's a sort of common thread there that anomaly detection is incredibly important because you want to find out, Helene, you also manage risk management. So the idea that spotting bad things fast is going to prevent bigger problems further down the line, I think. Yeah, the fraud detection example, that's relevant to just many businesses around the world. I'd like to hear some success stories like I don't know whether you can provide examples from specific clients, maybe more generally, just what sort of benefit can you expect from adopting real time analytics?

We worked with an industrial bakery whereby they were making the buns for a very well known global franchise of burger chains. We deployed a number of different sensors and therefore real time data off the back of those sensors looking at how to optimize the operations and how to risk manage better. And one of my favorite sayings about Internet things sensor data is it's going to tell us something we don't know. Because when you go around a factory or however it is as a human, there are certain things that you can physically see, but of course there are certain things that you can't see. When we deployed the devices and used the data, we were able to see that the H Vac, so the air conditioning unit was running at 95% humidity. That's not good. Can I just say. And we told the factory that this was an issue and you would be so surprised to know they did absolutely nothing about it. Now why this is interesting for me, multiple different ways is this is all about the data telling you something and actioning off the back of it. So guess what, go in and fix it. Because it's probably going to be quicker and it's going to be cheaper if you fix it. I said they didn't fix it, they didn't fix it. And the whole thing went and they had to replace the entire unit. So what would have been a. I don't know, a 20, 30,000 Euro loss ended up being 150,000 plus they had to stop the production lines. There was what we call consequent business interruption. So the quicker you go in and fix these type of things, the better, the least damage. The other thing that was a little bit scary was there was a particle detection sensor that was on their production line. They didn't know that it was being switched off for two hours every day. So can you imagine, again you've got these buns going through, could have little bits of metal in all sorts of things like that and that's obviously going into human consumption. And you have the potential for what they call a product recall loss. So again I go back to that source of senses plus data plus behavioral change. And the quicker that you do that, the least impact that you have. And everybody, nobody wants to have a claim, right? Nobody wants to have something happen. So can you actually use these more broadly and make a difference? Because you're reacting to data much, much more quickly.

That's a really interesting point that you can't just go and change your analytics stack but then not change the processes as well because otherwise you're not going to be able to take advantage of it. Okay, so maybe we'll just talk about who needs to be involved in this. Suppose your CEO goes okay, we need to data analytics, let's make it happen. Who needs to be involved in this transition? Which teams or roles need to be involved?

Absolutely. I'd say that every time I've seen it work out, the business units were heavily involved because the need usually comes from the business, from the use case and pretty much they need to own it. But then there's the critical question of what sort of technologies would we need depending on our latency requirements and functional requirements. The traditional way of looking at the real time projects would be that you'd have data scientists drafting a project, let's say in Jupyter's. But the intuition is that that would Be batch, right? And then this is often given to somebody who would be a flink expert or a real time expert. It would be one person on the team whose job would be to, sorry, translate this one into a real time setting, hoping that all the functionalities would carry through. This is usually not the case because actually the real time stack has limitations in terms of what is possible to express and there are certain issues with consistency of how the results come back. So often translation is either difficult or makes the use case a bit crippled compared to what the data scientists designed. And then these are data engineers who usually translate the use case into real time and then this is being run, the pipeline is being run right and maintained for this case. So you usually have the business owner, let's say the business of the use case, data scientists or data science team that designs the use case and the solution, and data engineers whose job is to make it work in real time. Now there are actually smarter ways to do it. There are possibilities for data scientists to be designing for real time already, such that we would lose this moment of translation from batch to real time. Then real time becomes more like pressing a switch. Of course there's a deployment question, but you pretty much can start working even Jupyter's on real time data and having replay and all data science work done on live data if you wish. It's actually right now possible. So we're also observing a certain shift where data science people start to be getting into real time just because they have to deal with updating, changing data. And personally, by the way, I rarely talk about real time. I like to talk just about the fact that every data is changing. So if you wish we could push it to saying that every data is real time or every data is, I say incremental because you have incremental changes that you need and updates that you need to take care of if you're dealing with a living system. And it's not just a historical analysis, the big question is how to deal with data updates and then making sure that they're taken into account properly. But in terms of teams, you have your data scientists, data engineering and let's say some change management that is also happening right now at organizations. But for those who didn't have streaming talent before, this is a good news because all of a sudden, because your entire data team, assuming they work in Python, they can actually deliver operational analytics that you care about. So they can deliver the ROIs, help you protect your resources and make smarter decisions faster. Probably this is it. And then we've always seen more and more of business people being involved.

Obviously, I agree with what you're saying. I think what's been quite interesting in the experiences I've had around data scientists and business people together is it's really rare to find a data scientist who also has not only deep data domain knowledge, but also business domain knowledge that they understand sufficiently the use cases in order to deliver the business outcomes that are desired. And so that kind of the ability to kind of maybe have a translator type way role between the two, you know enough about each topic to be dangerous. Not necessarily the depth, but more of that generalist skill set so that you can make those connections. And I think what was super interesting for us again in insurance, I'm in risk. We had different types of business stakeholders. So we had one that was an underwriter. So an underwriter is somebody who assesses the risk and puts a price on it. You then had a risk manager and they're the folks about risk engineering that go to sites and see and so forth. And it was fascinating the kind of conversation and dynamic between the two because it's as much about translating the fire hose of data that you get the noise to what are the insights and the interesting elements from the data that make sense for the use case. And I think we had a sort of real time data around, I can't remember exactly something like 300 data points every however many seconds. Only 60 of those were remotely interesting and satisfied the use case. So again, you couldn't have had that outcome a without translators in inverted commas. But also different business people looking at the different angles of the use case.

Yeah, it's a perennial challenge, I think, is getting data people to talk to business people. It's something we worry a lot about on this show.

Actually, could I comment on this? Because I've seen it play out wonderfully. Actually one of our customers was like a more traditional one, but this goes back to their entire digital transformation. So at certain point they took people from operations and upskilled them to data science, which means that they came and they are able to go to, let's say the youngsters in operations right now and say, sorry, I know how this is being delivered or I know how this process works. Right. So I know how to optimize it. And they come with just such an authority and knowledge when it comes to really operations and what matters, it actually works. Then they can innovate. Right. Very fast.

That's brilliant. I love the idea of just taking your business people, your operations people, giving them some data skills and then suddenly magic happens. Yeah, it's a wonderful story. All right, I think both of you agreed that if you're going to do this right, you need some data people, you need some infrastructure people, you need some business people. And that leads to a question of like, which one of your executives needs to be in charge? Is it going to be like your chief data officer, your chief information officer, your chief revenue officer, who is going to be the person accountable for running a real time analytics program?

In practice, I've usually seen the chief data officer. Actually the more successful ones are driven by business units that have the pay. But it depends sometimes on what sort of ambition the company has. So there are beautiful examples of companies that just move to real time. They move to real time data warehouse. You're moving out of Oracle and you could possibly just have your data warehouse organized in Kafka. And we have like, even though we're not this critical latency type of business, you just assume that all data is flowing and you have your data warehouse. Right. And then this is pretty much a CIO who makes a decision. So these are the big ones where we're more about constructing the full stack. And this is where, of course this is driven by the CIO and CBOs. But then often the way we see real time, you know, sneaking into organizations is we more business units that have the pain. It's like I'm losing this much money because I don't know how to react. I'm operating in the blind right on this use case.

Okay. So it's going to be your business units or your business leaders that request this. And then it's going to be between the data team and the IT team to figure out how to implement it.

This is where we usually see it happening faster unless there is a full transformation of the stack. And also we believe that it's in a way smarter to look up the use case and deliver the use case, especially right now. And speed matters also for the market share. With the arrival of LLMs right at Pathway, we believe that almost every pipeline will become an LLM pipeline in the future. As we used to be planning ML models in the pipelines. I mean, right now we'll be plugging LLMs more and more and the speed of adoption matters a lot. Many rules of the game are broken and they're being reinvented right now. So we're looking at actually people redelivering use case by use case, ensuring adoption as well of the business users rather than waiting, you know, 10 years for a stack to be built. So if you think I'm exaggerating with the 10 years, but it was also like the go to market strategy of Snowflake. Instead of offering big data warehouse, it'll take millions. Business divisions were able to get going with their data from day one, paying easily.

I can certainly see how ripping everything out and going, okay, we're going to start again with a new data warehouse on the whole company level is difficult. So maybe doing it one business unit at a time is easier. Sorry, Helene, you wanted to add to that.

Yeah, I was going to kind of build on that comment. Some of those absolutely resonate with me as well. And it's either where the biggest problem to solve is or those that are more excited about trying to do something differently. Yes, if you've got a big data technology digital transformation strategy, but sometimes almost the steps before you get there are who is most excited to do something differently is that again, I'm bringing that human element in. They're willing to experiment, they're willing to get things wrong, they're willing to drive and learn what the benefits are. That may not necessarily be obvious. If it's something completely new. Right. What is real time data? What are the benefits? Well, go learn, go investigate, go trial. And then from a culture perspective, that kind of creates a bit of fomo, potentially fear of missing out of like, oh, hold on a minute, that team over there has managed to do that. And that's when you then can kind of elevate it potentially into being much, much more of a big transformation. But sometimes starting small, experimenting, learning with the right people and the right attitudes, then you can drive that broader scale. Because one of my mantras is I often say we don't have a tech or a data problem, we have an adoption problem. So how do you get people to want to learn, to upskill? To your point previously, how do you get people to want to do things differently? Because it's very easy to stay in a little comfort zone bubble, particularly if you're making money. Why change?

Do you want to expand on that? Do you have any tips for how you can get people to adopt new technology and techniques and actually go about doing the upskilling to make sure they can take advantage of new technology?

Oddly, it's about the relationships that you build within the business and who, you know, who are those people that are going to queue up right at 6am when Apple opened their doors because they're launching a new iPhone? Find those people because they're interested in dynamic anyway, and they're willing, they have that personality type that want to do something differently. I remember having a conversation with my board one time where I actually ended up doing a very, very big piece of work with quite a small team with significant benefits. And my board said, why did you work with that team? They're tiny. Why didn't you work with this team? They've got a much, much bigger revenue and it's a bit like they didn't want to. So find the people, those gems of people in your organization that have already got that right attitude and want to try something different because it's a real hard slog sometimes if it's the bigger team or the bigger thing and they're just like, no, we're making money, we don't need to do anything different. Find the pockets and then drive even more benefit. And that's when you get kind of excitement and new stories and benefits and then you get some something that's kind of that ground up swell of excitement within an organization.

I love that idea of just find your early adopters, find your champions and they're going to generate some excitement and hopefully can spread across the rest of your organization. So I'd like to know what's a good first project? You mentioned it's probably going to be one business division at a time or one team at a time. Are there any sort of easy or common, simple real time projects that you might want to take on first?

Absolutely. It may depend a bit on the adoption strategy that you have, of course. So there is one way that we show, depending on whom you're approaching, who wants to adopt, it may be the tech teams or it may be the business teams. Right. So in the case of business teams, the easiest ones to test start doing things on your own. Is anything linked to logs, logs monitoring. Right. Anomaly detection, pretty much management of your own work that is very fast, easy actually to test. And this is lovely. But then with business units, well, questions where it's the biggest pain and well, right now we actually do live in the AI world. So these are projects that can very quickly resolve issues of people and pretty massively like deep frustrations that we've seen. But the one that we really like is anything that's linked to productivity, such as people actually see the productivity gain and some of their frustrations are being taken away. And then of course there's an EROI behind it immediately. Because if you have, for example, 47% of digital workers who claim they are wasting time on just finding information, then assume you save like One hour per week for them at the level of, let's say Ericsson. Like these are big tens of millions, right? Just this. And I think one hour for searching for information is an underestimate on how much we're actually wasting. So the case that we really like that resonates well, it doesn't scream real time, however, you do have real time updates and all the documents and knowledge bases and stuff. Hence you actually need to be able to manage this. The one that we like is certain slides like, you know, I'm Bob. And then, you know, I think that Alice had this slide about a user growth maybe or maybe somebody on her team. And I don't know what it or that there was some sort of conference, how do you find it? And maybe there were other versions, right. And I need it right now for my work. And the usual chain would be you would go and ask kind of Alice, Alice would scratch her head, she would say no, it was Katie, you know, ask her, maybe it was her collaborator. And then it would go and run around like a chicken in my head and try to find that one data point or to speed up your work. And with a nice write down system that combines llands with updates, like all the nice indexing, you can get what you want immediately. And this is a very nice use case which leverages the fact that data is changing private data sets in such a way that it immediately fixes your life. So it's maybe one example that actually we get pathway see right now a lot. It's a simple one. And I needed rebuild it because I needed it. I imagine like everybody can need it in a way or needed it in their life. So maybe the larger story here is just find the use case that's just so obvious that it's resolving a little pain. And then of course if you move to big pipelines then it's a bit easier to show ROI because sometimes you can link the ROI to latency gain, which is an obvious case. And fraud detection in certain cases. Fraud detection can even open new revenue streams. Of course, explaining things with money makes it way easier.

I like the idea of just starting with thinking where am I wasting time at work? And what's annoying me? And those are the cases where you need to get faster and optimize things. All right, Elaine, do you have any more examples of good, easier projects to get started with?

I think about. So Susanna had some fantastic examples there which were perhaps more internally focused. Absolutely brilliant. But there's also the external lens. So two way. One is who are Your competitors and what are they doing and do you need to keep up or go faster? But also what are the demands of the client base that you're serving? So are they wanting more from you? Are they asking questions that you know, to your point, Susanna, oh, my God, it's going to take me three hours to answer this. But you need to answer those questions much, much more quickly. So often those external drivers from your client base make meaningful impact. I mean, the best projects I did were because a client wanted me to do something differently and I could then go internally and go, hey, we're going to lose this business if we don't do this, this and this. So you have to be a bit of a hustler, I think, in some, you know, go hustle for use cases, hustle for the doors that are going to be slightly open, that you can push open a bit more. So, yeah, always client base and also competition. So see who's doing what to drive forward.

I'd say it's a lot about the market share. So if you're worried about your market share, you can lose it not responding to customers or you can lose it just because you are unable to automate and move as fast as your competitors. So both internal and external. And this actually glues very nicely with AI. I mean, if you're able to make better decisions faster and serve your customers faster and better, then you're probably winning.

I think that's just incredibly important competitive advantage, just being able to do things more efficiently and faster than your competitors. So, yeah, that seems like a good sort of heuristic for deciding what projects to take on.

Oh, I challenged that a little bit.

Ooh, okay.

Not just speed, but it's the informed decision making at speed, I think. So can you make better, more informed decisions quicker than your competitors speed? If you're telling them a rubbish answer, I could quite easily say no to you. And that's really quick. And I've not put any thought into that. Whereas actually, if I can come up with something really informed, you need to do this, this and this, and this would make you safer, drive better, whatever. For you, I think it's the informed decision making at speed.

Actually, that's true. I can always write a data analysis that gives the wrong answer in like 0 seconds. Actually, this leads to a tricky question for you. How do you go from, okay, we've got a real time analytics capability to we're making good decisions in real time? Elaine, since you brought this up, this one's for you.

That's my payback isn't it? For challenging you? That is quite tricky because when you start small and obviously need to scale, you then start hitting. You probably need business model change as well. So I'll give you a good example of this within insurance, within if you're wanting to put a price on how much the insurance should be premium wise for building, you look at what typically are static data points. So you look at how old the building is, what it's made of, how many stories it's got, where it is, blah, blah. That doesn't change every five minutes, does it? Whereas now, because buildings have so many sensors and it's constantly giving you data about the building, how, you know, we all go into meeting rooms and we become too cold or too hot and we're changing the temperature and all those sorts of things actually that starts to make a difference. Building occupancy, indoor air quality, all sorts of things within the building. It is really difficult for an actuary to change their pricing model from 6 or 7, however many data points for premium to 60 plus data points every seconds, few minutes. Really, really difficult. There is one organization that's done it and they have driven so much more profitability because they made the business model change because otherwise it just stays as a surface or we're tinkering with it to driving real business value because they're able to put the construct around, being able to use the data and make those. As we talked about a moment ago, there's better, more informed decisions with better pricing that's more pertinent to that risk because they've got more information.

Okay, so I think this is back to we need the data teams to talk to the business teams again because what are the business teams going to do with all that new data that they now have? All right, so maybe we'll break this down. I think I promised we were going to talk more about processes beforehand. So Susanna, do you want to talk through what some of the process changes are going to be? So once you've got this real time analytics capability, how do you change your processes in order to get these new business models or otherwise take advantage of the technology?

Of course. I mean I would reverse it a bit in the sense that I would start the case, like the entire real time case from the business questions and needs that are there, then it's way easier because technology responds to an actual business use case. Right. So it depends again who's driving it. If it's being driven by detective visions or it's being driven by the business people who Own the use case. It is a bit easier when there is a business leader who knows, okay, we need to change our business model to become more competitive. And let's for example, try to adjust pricing. And how we do it is by going down and down in the tech stack and doing this. If, I don't know, maybe need Kafka, maybe we don't need Kafka. We can just have a fast Esprit and then smart analytics and stuff. So I would actually normally reverse the thinking and not say, okay, now I have real time, so what do I do about this? I would rather think about what I want to accomplish and then go down the real time path. One thing to remember is that real time is not as scary as it used to be. Many people thought, okay, I can't afford having, for example, I know real time pricing because that would be extremely expensive and I'm not even sure if I would ever get into it. And then talent is difficult to get on board, et cetera, et cetera. I mean, right now this is not the case anymore. But in terms of process change, you do have business models that are different. Normally it's driven by product people. So any time I've seen real time being adopted, we often talk about business people who are in the use case, but they're the product people actually drive the adoption and adjust the little elements. For example, how information is delivered to make it useful. So we very often see that it's important to have a simple way of prototyping how insights are delivered to users and depending on their level in their organization. And this is actually a very powerful tool for data people who work on the use case is to find a way, okay, I'm not just doing my analysis right, but I need to find a way in which this insight will be meaningful to the business user. And depending on levels of users is maybe very different. So we talked about logistics a bit before. Like imagine you have somebody who's running a warehouse and this time it's not a data warehouse, it is an actual warehouse, right? That person probably doesn't need an interactive dashboard with thousands of different parameters. That person may actually need text with an alert about the fact that this sort of truck is going to arrive within two hours or not, because then there are some decisions that are to be made depending on this information. But getting to this like level of granularity of how specific insights should be delivered to whom is actually a big thing. There's a lot of usability elements also in the ways we can consume analytics and it links to real time. It's true for any data science, but it's especially true to real time because you have more dynamic ways of delivering information like alerts. And if your alerts are bad, you will generate habits which are actually very, very negative because people will just start ignoring your alerts. Hence you need to make sure you're not generating false positives because then the entire real time system just doesn't make any sense. And this used to be actually a very big pain of the real time technology that it was tough to reach consistency. So you would kind of end up with a light that's just flashing like an alert light. And then what you've seen really is that it was smart to go with some sort of dashboard prototypes to business. People can nail what sort of data is one thing, but then insight is a completely different one. And then how we deliver the insight is like a third step. So first thing is like having a simple way to prototype this, including colors and usability, like a small thing of the like. If you're able to share your link to your dashboard and the dashboard stays the same with all the filters, this is extremely useful because you can share it with a colleague and just say hey listen, like this is wrong or something is up, what can we do? Or you share it with your boss and you have a report. This actually speeds up things enormously. And there's a very, very simple thing. And once you get this, you can optimize other ways in which you deliver, like I mentioned, texts, sliders, whatever is used really in the organization to be able to consume the outputs of a real time system. This is for the somehow human in the loop scenarios. Of course there are many non human in the loop scenarios, but then we would be talking of for detection or trading, et cetera.

I like the idea of just going beyond oh, the output, the analytics is a dashboard actually sometimes people want an alert and yeah, just sending them a text or message. Well, plenty of different messaging systems, but just getting the insights to people where they need it immediately in a simple format that can be very powerful. We talked a bit about how processes can change. I'd also like to talk about how does organizational structure change? So do you need a different team setup?

I'd say if I can take it for really for real time. Talking from infrastructure kind of data and infrastructure perspective, there's a strong trend that we're seeing right now of so called modern data teams where data engineering is disappearing. So we're actually having let's say full stack data people and you actually have to merge software engineering with data skills. And then of course you would have DevOps, you would have people for deployment, et cetera, but the more modern teams that you would see out there right now, and let's say the digital native companies start to just not having data engineers as a separate. The assumption is that if you're, if you're actually especially true with AI because you need to be able to control the pipeline, because any small change at any step of the pipeline can actually impact the way your model is operating. So you need to be able to have an end to end view and control over the deployment. So the data engineers either need to kind of upscale or like stay in specific use cases. This is one thing that we see. And then there is a bigger shift which is happening of course right now with LLMs. So LLMs are adopted mostly by the data teams or there are new teams being created. Right. It's a bit shaky still because everybody needs to adopt. But it's very disruptive in fact to the way the organization is working. And here, you know, you're merging like all these skills together. But in the end it's the question, okay, who's delivering the pipeline? Who's responsible for this pipeline? Who's responsible not just for the pipeline to work? And who's getting the calls in the middle of the night that hey, it's broken. Which would be the data engineer's old job sometimes, but right now it's more about is this model drifting right? Who's taking responsibility for the output of the models? And this is becoming a bit tricky in terms of how integrated teams need to be at this stage.

That's interesting that data engineering and software engineering are merging, I guess because there's a lot of software involved in building these pipelines. I'm curious as to whether the data analyst role is changing as well. Like, is a real time data analyst different from a regular traditional data analyst? Is that job changing?

I can't answer that question, but I'm just reflecting on what Susannah said because I had in my head that by having real time data you in some respects need to be more specialist. But actually when I listen to Susanna, it sounds like your teams are becoming more generalist, as in they need to know more than one thing, whereas I had in my head actually, because you've now got greater insight, it becomes much more of an ecosystem from an organizational perspective where you have more specialist roles actually. So it is really interesting kind of that contrast of how do you get the data, how do you get the insight and then what do you do from a behavioral perspective once you've got the insights? So I saw that sort of those series of roles because to your point, when somebody sends an alert, in my world it would be a tap drip. So I need to go and get a plumber. Well, that's not an insurer, that's a very specialist role. Or your temperature starts to get too hot, you're going to again send an electrician in, for example. So I had it in my head that it was an ecosystem with specialization. But I'm really interested because you said.

The opposite for how you act on the information, of course, because then it impacts the business roles. But as for the data team, let's say how it was designed, right now this is pretty much it's even the thing of 2023, more than 2024. The rise of the modern data teams and LLMs are changing this, adding yet another layer to the level of even complexity. It's just becoming different. I think in many ways it's becoming easier because tooling evolved. Indeed, back in the days we'd have one, two people specialized in streaming. I can take a case of like one of the F1 racing teams, right. So there would be in the past, let's say some dedicated folks to streaming and others would be doing batch analytics. If all of a sudden everybody can just do things in real time from day one, it's just way easier and cheaper. Right? But before real time data pipelines were pretty hermetic also for just technical reasons, or I call it linguistic because it's a question of language. Right. There was no lingua franca between batch and realtime. Real time was traditionally more built in Java and this sort of technologies. Whereas the dominant language for data scientists, data analysts is Python and SQL. But especially right now with AI common, we have to be working in Python. This is the one thing we need to be doing. Of course, Python is not dramatically efficient, so making it real time is actually pretty funny. Of course it's not Python that starts to run in real time. There are ways to allow people, data analysts, data scientists, to work with Python and still making it run in real time. It's actually one of the things we did at Pathway. The engine is written in Rust and then everything's being executed on the Rust engine, which speeds things up. But the interface, the way one interacts with real time pipelines is in Python. So you can code your thing in Python. It's nice and I'd say fluffy in a certain way or customary for the data professionals. But then you get the speed and you get the ability to actually run in real time. But it's not just us. Actually the thing about the data modern data teams is something I've heard numerous times from some most innovative companies. Of course, is there probably the scale ups that are driving this? It's like it's definitely not the case of Walmart for example, yet perhaps more.

Insurance because we're very old.

But it is changing because sometimes you just need an external event to shake things up. And the arrival of LLMs started to make it okay to break all the rules in data organization all of a sudden people for example, who are afraid of bringing somebody in, getting help, like somebody would be saying, no, I'm building a big stack and I have a plan for five years of this. And they're like, okay, I don't know what's happening. We have to know where to make the bets. We need to move fast, so let's act and move faster. And then yes, people need to be more aware of the full implications of the deployments. So both development and deployment because sorry, I'm biased towards elements right now because this is like most of actually the user pathway right now is for llabs, but this is also something that we're going to be seeing everywhere. So what you see now is that it's very tough to put such systems in production because of the disconnect between talent in a way and sometimes an organization and then also development of technology. It's pretty easy to make a nice use case or like a prototype to test with users, which is great, but they need to put it at scale. And this includes data updates, real time data. This includes serving the model to the user's privacy, making sure that all of a sudden the answers that are maybe should be restricted to the eyes of the CEO are not given to any employee because that would be tricky. And all of this actually requires a sensitivity and like a way closer collaboration between different members of the data team. So of course there are still people who are more specialized in this and that, like who have better experience, but they should be actually merging more and more like software data skills. And by data I would extend it to LLMs including deployments.

So maybe a few people have to be go very deep on particular areas but actually in general everyone needs to learn a lot about several different roles. So lots more learning and upskilling for everyone it sounds like.

Yeah, but then you also get automation of certain tests so I'll be able to perform them faster. So the second, let's say part of LLMs is that we'll get certain tasks being done faster or automated as the value will be in the ability to think systemic, like in terms of systems. Right. Rather than point wise.

So I do love a good disaster story. So I'd like to know what are the common mistakes organizations make when trying to adopt real time analytics? Helene, I'm sure you must have some good disaster stories from the insurance industry.

Oh, disaster stories. Surely those should be called learnings. No, yes, Lessons learned, Lessons learned. I mean for me I kind of veer away from actually the day it's all about the people. It's always all about the people getting the people to make sure that you get the right requirements that you define the use case. You get the people that are going to actually want to be involved. We haven't talked about political capital. I mean I've had so many failures from initiatives because the leader who was championing it and so excited then left the organization. And of course a new leader doesn't want somebody else's idea, they want to then build their own. So unfortunately many, many things stopped because of that. So yeah, I can't really talk to kind of data specifics. For me I always. Disasters are always around the people in the broadest sense because you, you miss something people don't like change or what have you.

All right, so it sounds like the organizational side of things are very important, like make sure you do your planning properly, you've got a real use case and you've got executive buy in.

Correct.

Okay, Susannah, any more lessons learned?

Yeah, I would agree with this. Just generally speaking, namely use case first. Although of course like sometimes you want to educate a bit. Right? Especially when data team wants to educate the business and they would start and so maybe it's a learning, maybe it has to be happening just at certain points and breaks the wall. Actually I'd say that especially let's say two years ago I heard more horror stories about adopting real time data pipelines than I heard like good stories. And that was mostly linked to the difficulty of getting building the stack. And again this depends on our definition of real time because there is a question of being in the real time streaming data stack, like technological stack, this is one thing. And then there's just a question of having pretty much operational analytics or you know, data flowing at latency that makes sense for your business. I would separate those two. But it's true that getting into for example, you know, the Kafka ecosystem, people are saying I don't want to babysit Kafka. I just, I was just talking with a fintech and they said we know Kafka very well. We know it so well that we didn't want it. That was honestly, this is a like 100%. I'm not even swapping it for storytelling. So this is Kafka, right? And then usually in the streaming we have some sort of message broker would be Kafka, the most traditional one. Then you have stream processing or some sort of framework to do analytics or to actually be able to process this data and get the output and latency that you require. And I'd say that I've seen people usually embarking on big projects that would last nine months and often fail for numerous reasons. I mean one would be for example, trying to translate a use case which was prototyped in batch into real time and running into just limitations of the streaming technology. And this happened more often than not. Or folks who are trying to have a hybrid architecture. And in that hybrid architecture, I mean this is becoming so expensive to run and maintain. So you have some bits of real time, some bits of badge or like kind of doing both. And then with certain vendors when you're starting to do this, it becomes prohibitive in terms of cost. And then unfortunately this goes back to the use case, right? Because question is, okay, is it really affordable for us? I mean, is it so interesting from the business standpoint that we want to be doing it? So of course lowering complexity and lowering the costs opens up those use cases more easily.

Okay, so you really got to think about your tech stack before you spend a year building something new out and then discovering it doesn't work.

And actually there are, right now there are nice possibilities. You don't need to get into Kafka, for example, if you want to have low latency analytics. I mean for some people it's enough to have a fast updating storage like a fast S3 and then plugin analytics framework like us actually. And you still can get in many cases sub second latency of course depends on what you're doing. But point is you want your insights faster. In many cases you can probably just reduce this complexity by choosing a stack which is a bit different still. And it's not streaming in the tech stack sense. It still can be sort of streaming in just the way that it delivers your use case.

I'd flip a couple of points around actually, and one of them is probably more specific because I'm in a very highly regulated environment. But one of them is simple kind of innovation management. So one regulation, so one of the things that actually enabled me to be successful was to bring a cross section of individuals from the organization that were in the government's legal audit teams. And we presented every single one of our pipeline and experiments before we kind of got into scale or anything like that to that group. And it was very, very deliberately done because we wanted to hear every question from that group because some of it was new technology, new data points, real time data and so forth. And so they needed to know about it straight away and learn from each other. So audit might ask one question versus compliance versus blah, blah, blah. So you were able to kind of build it by design to be compliant with the regulated needs that we had. And that, oh my God, we went slow, but we went so much faster as a result. And then the other one was simple stage gates. So you talk designer about kind of a nine month project, you may have had these. So I don't know. But just having a series of stage gates, it must be able to do this before we're going to invest more time, money or resource into the next stage. And being really deliberate about how you move from A to B to C to D and stop things that are not going to work far sooner because people get this idea we've got to deliver the use case. Well, do you have points? Where is it still doing this? Do we need to pivot? And it's about bringing different kind of those innovation management tools and techniques to bear that are different to project management. Whether you are agile, wagile, waterfall or whatever else. It is actually having something a little bit more disciplined in terms of those innovation structures to stop things much, much more quickly and therefore learn to stop them becoming real disasters.

I mean the nine months I was referring to three from a technical perspective, how long does it take you to write the code? Pretty much. And it was traditionally the timeline.

All right, yeah, so it does sound like again, we're back to really speak to a lot of people in your organization, make sure your plans make sense and you've got a way to exit the project gracefully if things aren't going as you planned. All right, super. So just to wrap up, what are you most excited about in the world of real time analytics?

So I have recently started to learn about digital twins. So all sorts and all manner of different types of digital twins, which I think the exciting part for me is the visualization of the data and being able to manipulate it in a digital environment without affecting the physical thing that you're emulating. In a virtual way, but kind of the blend of IoT sensors, cloud technology, big data, every buzzword we can play buzzword bingo, AI and kind of the metaverse, extended reality, bringing all those things together to create a visualization. And you said something earlier, Susanna, that struck me when you were talking about the layout of a warehouse. So BMW have got a digital twin of their warehouse which I've had a look at, which is crazy clever, but even something as simple as if a worker moves 10 centimeters, they can make x percent improvement in the throughput through their factory. And they only knew that through a digital twin. So that's what I'm excited about, you know, novice very much. But I'm upskilling myself. There you go.

That's very cool. Actually, you might be the first person this year that's been very positive about the idea of a metaverse. But yeah, certainly digital twins, very, very cool technology. And yeah, a lot of applications of that. If you've got physical products or warehouses or whatever, wonderful stuff. All right, Susanna.

Yeah, I would be a bit more forward looking. I'm extremely excited about the potential of live AI. So we don't have it right now. Right. LLMs are batch by definition, but being able to actually act and have AI systems operating on fresh facts, but like we do, this is going to be enormous breakthrough and this is something that at Pathway we're extremely excited about.

All right, real time AI, that's maybe the next episode and we'll have to come back to that. But also a very cool idea. All right, wonderful with that. Thank you so much, Susanna. Thank you so much, Elaine. Real pleasure chatting to both of you.

Likewise. Thank you so much for having us.

Thank you.