#189 From BI to AI with Nick Magnuson, Head of AI at Qlik

The intersection of structured and unstructured data to me is really interesting and we're doing some very, I think, innovative and interesting things in there to help try and exploit the opportunities that those two seemingly isolated sets of data have, can provide. And, you know, foundational models are at the heart of that. You know, generative AI is at the heart of that.

Hi Nick, welcome to the show.

Thanks, Richie. Brilliant.

So I'd love to dive straight in and talk about what's going on. Qlik. So I think Qlik, Sense is perhaps your most famous product. This is your business intelligence platform. So since there are a lot of BI platforms around at the moment. So can you tell me what's special about Sense?

Yeah, Sense is sort of our core product. And actually, Richie, this is the 30 year mark on Qlik's anniversary, so we're celebrating the long history of Qlik in the analytics and data space. Sense is a product that enables people to get insight out of their data with relative ease. That can be a very cumbersome process. So that could be looking at very complex data across a variety of different sources that you need to bring together, need to combine it and need to be able to make sense of it and then explore it through a visual, visual means. So census, sort of the core product that makes that process a lot easier, enables a lot of other people that aren't really sort of the SQL ninjas of the world to be able to do that. And yeah, it's been a core product. Core product from us since its inception. Okay.

So it's really about the data analytics side of things. But Sense is part of the largest suite of things. So can you talk me through like what the whole suite involves?

Yeah, certainly. So, yeah, like Sensus, the core analytics product today, building on QlikView, which many viewers and listeners may be more accustomed, that's the longer history product on the analytics side. But over the last decade or so, Qlik has made, I think, a pretty distinct effort to try and expand that core offering into particularly into the data integration side of the data analytics workflow so that you can bring in sources from a variety of different, both on prem and cloud and hybrid sources and do that at scale. Right. So doing it with an enterprise grade data pipeline and capability to manage those workloads, including things like change, data capture, so that you're incrementally updating that information and then building pipelines that can transform it into something that's usable. Obviously that plays very well with the analytics side where you're then taking that data and Then making robust analysis and hopefully making decisions off of it. And then even more recently we've added capabilities that are near and dear to my heart around machine learning and AI as well as automations. The ability to take data as opposed to read and infer it and then take an action and do that programmatically through an automation. And then of course, just this year we announced the acquisition of Talend. So Talend brings a very large portfolio of different products all the way from data integration into data quality, stewardship, inventory and some robust capabilities around data prep. So at the end of the day, like the portfolio has expanded quite substantially into an end to end platform that allows our customers to be empowered to make use of that data at any stage of that whole workflow. So it's a pretty exciting product portfolio that we've now assembled and put together.

So this for every sort of stage of the data workflow, that's kind of nice. We'll leave all this stuff on data preparation, the data engineering bits, maybe for later. Let's get into the juicy stuff I think with machine learning and AI. So I know you've just launched Stage, so that's your new AI product. Go on, talk me through what the point of this is and how it fits into the rest of your AI tools.

Yeah, Stage is an interesting announcement, right? I don't actually think it's as much of a product as it is a strategy. It's a strategy that helps our customers be successful in the implementation of their own AI. And for me that comes down to three different pillars. The first, which is establishing a data foundation that you can leverage to build out AI models. And that data foundation is fairly pivotal because the whole garbage in, garbage out is no less true than it is with AI where you've got to have the highest level of data integrity so that you can trust the outputs that come from AI, because the outputs from AI are dependent completely on the inputs. So that Data foundation is that first pillar to enable that success. The second piece is ensuring that we as an organization use AI to help our customers by infusing it into those capabilities. So as you're engaging in any part of that data analytics workflow, there's AI there to help make that process faster, easier, more efficient, et cetera. And then the last piece is more about recognizing that customers want to get hands on, they want to build their own AI solutions, they want to bring their data to that, and we need self service AI to help them support that. So we're building new products and we do have products today that help customers realize that objective. So that stage in a nutshell, again, I think the idea is to help our customers get to successful implementation of AI and you know, that is a multifaceted effort.

I do think that's fascinating that you said that getting success from AI requires you to have a strong data foundation to begin with. And I'd love to get into that more in depth throughout this episode. Before we get to that, I think certainly datacamp has a lot of customers where they say, okay, we know we need to do something involving AI, but we're new to this, we're not quite sure what to do. So can you perhaps talk about what some of the most common AI use cases are from QLIK customers?

Yeah, I can speak specifically on the newer forms of AI with generative because I think that's where a lot of the focus is today. And I think one thing to recognize here is that everyone's exploring. I talked to our global system integrators, our biggest partners across the globe, and they're doing tons of POCs, tons of POCs, but very little is in production. It just gives you a sense of where people are at. They're trying to understand this technology, how do, how do we use it, what kind of things can it solve for? And so there's a number of ways to look at that. I look at it from the perspective of, you know, what, what can actually do, what can generative AI do? And I think there's three things that at least at this stage of the technology's evolution is very clear. They can do it can summarize very, very efficiently, it can create content, and some might argue not good content, but it can create content in a very efficient manner. And then the third way that I've seen it implemented that I think is worth noting is just around code generation or code interpretation. Now those three things can be manifested in a lot of different ways. People look at chatbots. To me, chatbots are an implementation, not necessarily what the gen is doing. Gen is doing the summarization or content creation through that mechanism. But you know, that's sort of the primary viewpoint I take is what is it actually doing? And those three things are sort of the primary use cases that, that I've seen so far.

Are there any simple sort of high impact use cases you think are good as a first example for or a first project for enterprises to start off with?

Yeah, I love the in what I would call internal use cases. So using generative AI within the organization, not as a customer facing or some Sort of product offering. And the reason for that is, you know, if you think of the, the various functions you have within an organization, big or small, you've got, you know, you've got sales professionals that are in the market trying to position your, your service or your product and they're doing that based on a knowledge of what your product does, how it operates. You think of the people that are supporting customers, again they're, they're interacting with unstructured data to help them understand and how to communicate, troubleshoot with customers. You know, so you basically go across the organization and almost every role is interacting with a ton of unstructured data, knowledge bases, et cetera, to help them do their job better, to become experts in the, you know, their given role. And the process of like interacting with that unstructured data can be very tedious because you've got to go and find a document or you got to go like the most common thing is someone pulls up Ms. Teams or Slack and says to their best friend, hey, where can I find this information? And that's not very efficient. So the use cases I like are, hey, take all that documentation and put it into a, you know, a reg architecture that an LLM can actually use so that when you have that question you can ask, you can ask a bot and they can go and reference that material and give you something of use both for the summarization and the content generation. And I like that because one, the value is clear, like you're just going to make your employees more efficient in what they're doing. So that's, that's fairly well established because they're, you know, there's documents that you already have. Like the relative level of effort to bring those solutions to bear, it's, it's not as onerous as it might seem. So again, I think you've got value and you've got relatively little, little level of effort. And then the third thing is if you don't get it perfect the first time, you know, these aren't mission critical business functions that if it doesn't work like, you know, you're out of business. This is, you know, stuff if you don't get right on the first time, there's low risk, low consequence. So I like those use cases, I think they naturally then lead into more sophisticated use cases. But it's a good entree, it's a good way to get started.

I really like the idea of doing things internally just while you're trying to figure things out because you are without doubt going to have like your first attempt at some kind of bot is going to say something pretty stupid. Better to say that to an employee rather than one of your most important customers. Definitely. And in general, do you think that enterprises have different requirements around AI compared to individuals?

Yes and no. I think there's commonality, like whether I'm sharing my information with an LLM or I'm doing on behalf of an organization, like I don't want to share certain information, I don't want to share sensitive information, confidential information. I think the parallel there is quite clear. Now, on the other hand, organizations have, I think, a higher burden on this because they're also working with their own end customers. And by proxy of doing that, they have information about their customers that potentially, you know, they're effectively the steward of that. And to share that with an LLM, I think creates a higher, a higher level of burden on the organization than it does on the individual, you know, So I think that that may be the biggest difference is that organizations work on behalf of other customers and so therefore they carry. Yeah, they carry a higher burden on that, absolutely.

So it seems like the big difference is really how much responsibility you have for things like data privacy, data security. So do you have any advice on how you might go about dealing with that?

Yeah, I mean, I think the biggest things there are putting in policies and procedures around this. I think every organization at this stage of the game should have documented what their expectations are both internally and then externally in terms of how they use generative AI and what sort of conditions surround that around privacy and security. For those that are looking for a bit of a guideline on that, the OAS just recently put out, I think, a pretty good set of recommendations and vulnerabilities with large language models in particular, that for me would be the blueprint for any organization in terms of how you want to account for those vulnerabilities, both from a security and privacy standpoint. But at the end of the day, most of it comes down to, you know, you can't share private information. Right. And I think that's, that's pretty common, pretty well understood, you know, for organizations you certainly don't want to be sharing confidential, trade secret or even, you know, IP like these, these large language models, they, they, they have a tough time unlearning what they've seen. So, you know, that is a, I think a key thing is you don't want to be sharing any of that. So there's a governance framework that needs to be in place to prevent that. And there's certain techniques you can use and way in which you interact with LLMs to make sure that that's a bit more secure.

So just related to this, I think a lot of organizations must have dealt with a lot of these similar issues when they've started moving things into the cloud. So using SaaS products, is it the same situation or do you think there are additional problems with large language models?

I think there are additional conditions with large language models that are new to the same set of conditions that you may have, you know, with going to the cloud or GDPR or you know, you name a data, data sovereignty and that is that the, you know, particularly the public LLMs like they're fairly black box in terms of, you know, what, how they were constructed, what they were trained on and the inputs. And so when you push a, you know, if you were to push any of yours like sensitive information to that, like realistically, you don't know exactly how that's going to be used versus if it's being pushed into the cloud. Okay, well, it's just, you know, it's in the cloud. I can, it's a little bit less transparent, if you will, as to, as to what the ramifications for any of that might be. Okay.

And when you're not quite sure what's going on, you probably need to be a little bit more risk averse in that case.

Yep.

Okay.

Yeah, I like to, I like to use the analogy that if, you know, you met someone for the first time, you're not going to divulge like all of your very secret information. You're probably going to talk about what, the weather. Everyone talks about the weather. And that's because it's a fairly benign topic. And I think the, you know, the approach here could be very similar where, you know, you're working with Genanic for the first time, you got to be very cautious about how you use it until you understand and start to build trust with the systems that, that are built on top of this new technology.

Okay, yeah, so it's like don't give your credit card number to someone you just met at the bus stop. Right, right, cool. So before you were talking about data quality and the importance of having good data in order to get good results from AI. So I'd like to go back to that again. So it does feel a bit like, well, everyone's using the same foundational LLMs. So really the, the benefits you get, you need better data because everyone's using the same AI. So can you talk to me a bit about how you get higher data quality across your organization.

Yeah, data is paramount. It's really critical. And I think I have, if not a unique, at least a well funded perspective on this. The first 15 years of my career I was an investor, specifically I was a quant, so a quantitative investor. I was using AI and ML long before data science was even a, a term that we all now know. And I went through the process of using that techniques to like find signal in the market and obviously try and take advantage of it. And to make a 15 year story short, we went through a very discreet evolution where at first nobody was using it, nobody could trust it, nobody was using machine learning. And then people started using, they started creating an advantage using it and then everybody was using it. And it became so ubiquitous that the math that supported the algorithmic trading and that sort of stuff was really not the differentiator. It became the data, it became the data. And so organizations were investing heavily in trying to scrape together whatever different data they could so that they had some unique market advantage. And I think the parallel is the same here. As businesses start to use AI very quickly, we will all be using the same foundational models, same prompt tuning, whatever the technique may be, and it'll actually come back down to what data do you have that's uniquely different than the competitor across the street? So in terms of data quality, there's a couple things that I think are super important. One is AI loves the variety of data. That's where it excels. It can see patterns that we can, it can find those, you know, those little tidbits of information that the human eye could never detect. So you need to bring a lot of different data, disparate data together and allow AI to kind of tackle it. That means like I said before, it could be on, on prem, could be hybrid, could be cloud. But organizations that are setting themselves up to collect as much data as possible I think are going to be in an advantage. The second piece of it, of course you can bring all that data together, but if it's, it's spotty, it's not cleansed, there's no quality, there's no governance around it that can make it virtually useless. So you've got to invest in data collection practices that make that data standardized and then therefore usable cleansing of that data should be standardized and where possible automated so that as the data comes in, it's in a form that people can use. And then I think the other piece on top of it is transparency. It's because data will come in and there'll be transformations that'll be combined with other. As someone who's using it as a end user to actually build a applications or AI models on top, you got to have an understanding of where that data came from. So lineage transparency, auditability, those things are also very important because you need to establish trust. And trust comes from an ability to understand, you know, and understand where that data came from. So that's a lot, obviously, but that's actually where I think the money is made is in investing in establishing that data foundation so that when you build AI, you can do it with conviction, you do it, confidence, it's trusted, well governed, et cetera.

Okay, you're right, that does sound like a lot. So you need lots of data and lots of different types of data, and you need to make sure it's all correct or suitable for use and well governed and all that kind of stuff. So, because that's just a, a massive amount of tasks, I mean, it's probably an entire podcast episode in itself. But can you just give me a quick overview of like, where do you get started with this? How do you just make those sort of incremental improvements in data quality or data quantity from where you are now?

Yeah, I think you pick out a specific use case. I love to start, you know, with something small in scope and make sure that small scope has the right data, the right, the right connectivity, the right data models that sit on top of it. And as you prove that out, then that becomes a launching point for a further investment to make that, you know, make that data fabric, if you will, of the same standard. Right, but you know, if you try and boil the ocean, like, I mean, I went through that list fairly purposely because it is a big undertaking, but that doesn't mean that you can't start with a small slice of it and then through that success, kind of employ that, employ that elsewhere. Okay.

Yeah. So really just pick a use case and then go for it. And just as a bullet to that. Is there like one specific use case you think is, is a good place to start? I know this is going to depend a lot on the business, but are there any sort of common themes here?

I don't, I, I don't necessarily think so. I think that's going to depend on, on each organization. I certainly would advise, and as we do with our customers, you know, start with a use case that will drive some business value. But you already kind of understand that there's a level of quality to the data that's needed to support it. Right. You don't want to go into a use case where, hey, this would be really cool. But I don't even know if we have the data. I don't even know if the data is of requisite quality or depth or whatever. So I think the marriage of a business case that has some value and it doesn't have to be like game changing, but it has to have value, but certainly is supported by, at least superficially, you know, that that data is there. You know, we're in a unique position at Qlik because a lot of that data that we're talking about is data that's already been used for, you know, standard purposes around visualization or reporting or stuff like that. So it's largely cleansed, it's largely prepared, it's largely known. That's a pretty good starting point for a lot of organizations because you're not trying to go on fetch new data that you've never seen before. It's data that you're already familiar with.

Okay. I mean that's the same reasonable like, you know, think of it about, yeah, business cases before you, before you start diving off on, on these new projects. All right, so I'd like to talk a bit about how you go about using generative AI or AI more generally. So I know there's a lot of people going to ChatGPT and sort of using these large language models sort of directly from the source. But then there's also a lot of companies building AI into their product. I mean, we're doing this at Data Camp. I know you're doing it at qlik and a lot of other companies are doing this. So when do you want to use an LLM directly versus have it built into a product?

Yeah, it's a good question. And I think that comes back again to the use case. So, you know, if you're looking for, you know, general responses, generic content, an out of the box solution can do the job. I use it, you know, fairly frequently. My kids use it quite frequently. Like it's, you know, everybody has cracked open GPT and that's part of the, part of the excitement is everyone knows like there's, there's real implication, real, real opportunity with it. But you know, if you go to, if you go to chat GBT and ask a specific business question, it's not grounded in your, in your business, it doesn't know anything about your data. So it's when you need to get specific that I think you need to work with LLMs directly and that's when all these Privacy and security things really, really come into play. But yeah, so it's more around use cases. If you're looking to build something that has, it would require specific knowledge that isn't generally part of the corpus that these models have been trained on. Then you've got to go and work with an LLM directly and use techniques like RAG or prompt engineering or what have you to make sure that the model is grounded in the contextual information that makes the solution useful.

Yeah, absolutely. That sort of resonates a bit with my own experience. So if I'm trying to write some sort of marketing type copy, then just throw it into ChatGPT, that's fine. And it's marketing stuff so it's going to be public anyway, so no privacy worries. But then if I'm doing something technical like I'm coding, I want to, I want a separate product. I don't want to necessarily do that within ChatGPT itself. That's interesting. Okay, and I guess more generally, how do you think about whether organizations should be buying existing AI services versus building them in house?

For me, AI is a portfolio play 100%. Now when I say portfolio play, I mean it's, you're not doing one or the other, you're doing both. Like the way to get scale out of AI is to have a portfolio of solutions that you can leverage across different Personas within your organization, across, across different contexts. And so it's a portfolio. So in house use cases, depending on the organization, depending on how sparse those resources are, I mean, some organizations don't even have data science teams and that's fine, that's just a recognition that you're going to be looking at other solutions. So in house use cases, depending on the resource availability, my recommendation would be to focus on things that are, that require a level of customization, a level of specificity in the inputs that only like, you know, the very, those very talented individuals would be able to affect. And then, you know, for use cases that are very, that are very core to the business, so to speak, that require a level of precision in their implementation and where the risks are potentially quite large if you don't get it just right, that to me is like you want to be, you want to be hands on, you want to be very, very specific in the delivery of that. Now that's not to say that that's the only use case that can drive value the other variety of use cases and we have customers that are, you know, deriving multi millions of dollars of savings or efficiency gains just through Low hanging fruit use cases that you can get through built in technology that is catered to an analytics team, not a data science team or something like that. So you know, I think it really depends on the organization, it depends on the use case that you're going after. But I do believe that most organizations should be following a broader portfolio. Like it shouldn't be. We're going to this part of the org for all our AI because that does not scale. That's the one thing that's, you know, over the last 10 plus years now that, that, that, that leads to failure. I have an interesting anecdote. I was talking to a manager at one of our customers the other day who was opining on an AI technology that they had just purchased. And one of his technical resources reported back to him that this AI technology could only do 85% of what he could do. And the manager was like, that's pretty good. So his, his recommendation was, well, that's good because now you can focus on the other 15 and make that, you know, even better. And I think that type of mentality is the right one where hey, if we can use technology to get, you know, 80% better at everything else that we're doing, that leaves us a lot of resource to focus on the things that, you know, we haven't yet perfected.

I like that he's only been outsourced 85% so yeah, that's good. All right, so it does seem like you need some kind of big AI strategy then just to make sure that you've got like all your portfolio comes together, especially if you're using external AI things. You've got some in house projects, a lot of different roles involved. So first of all, like, where should this strategy come from? Like who should be responsible for it?

Yeah, I hate to put the burden on this individual, but for me it's the CEO. It has to be the CEO. And it has to be the CEO, because the possibility that you could build an AI strategy from the ground up, grassroots style, you know, department by, I don't think is realistic. You know, you'll get an isolated pocket of people doing something successful over here, but they're not talking to people over here. And so like none of it will, will hang together at the end of the day and it won't look like a strategy that, you know, can really drive the level of value that you want out of it. So to me it's the CEO. To me it's, it's a push down, it's, you know, driven from the top. You know, the strategy that I've seen that have been, that have been successful, like you can literally feel it permeate the organization across every level. Everyone's aligned on it, everyone's excited by it because everyone wants to get involved in this. So for me it goes from the top for better or worse.

Okay. And I suppose if your CEO is interested in AI, which hopefully they are, you're all right. If they're not, and you've got sort of accesses, how do you go about sort of persuading senior management or your CEO that they ought to get going with AI?

It's a good question. If you have a CEO that hasn't kind of caught on to this, then I think that it could be a challenge. You know, one of the things that, for me anyway, this is the second time I've seen this movie, right? I, I, I saw the AutoML, the automated machine learning kind of rise up and that took a long time because that technology was a little bit hard to understand. Like you've got this data in these algorithms. Like you know, you needed, you needed like four different things. You needed understanding of the math, you needed the ability to code, you needed domain expertise and you needed to be able to move the data around like a SQL ninja. And CEOs couldn't get their head around that. So that was kind of like, and it built some critical mass over time. Generative AI is different. Like everyone can crack open ChatGPT and go oh boy, you know, like this is a big deal. So I think for a large, least cross section of CEOs like they're already on it because the boards are asking them about it. At the end of the day, if you are one of those few that the CEO either hasn't got on board or isn't pushing that from a top down perspective, I think you've got to work on a proof of concept, a proof of concept that the CEO can relate to so that they can understand like the implications of it and potentially position it as, you know, this is what everyone else is starting to do and if we're not on board, like we're starting to put ourselves in a position where we're going to have to play catch up. Not to put the fear into the CEO, but that's the reality is everyone right now, like I said before, is trying to figure out how to use this technology and as soon as they do it's going to create compet competitive advantage. So you know, I think it's incumbent upon people to push that into the CEO's lap and help them understand, you know, how they could, how they could implement it, how they can take advantage of it.

Okay, that seems pretty sensible. And perhaps if the CEO is still resistant, then probably dust off your resume and look elsewhere. Okay, so we've established the CEO has got to be sort of leading their strategic efforts, cheerleading stuff. What else or. Well, what does the, what else does the C suite need to do in order to make sure that your AI projects are successful?

I think there's two things that I would point to. One is at the C level, you've got to make sure that everyone else within the organization feels empowered to use this technology, to apply it, to use cases that they're going to know very specifically because they're experts in that part of the organization or the domain. But they need to feel empowered, they need to have the technology, they need to have the resourcing to be able to do that. And then the second thing is, while they may feel empowered, they also can't go rogue and kind of use this without any guardrails. So the other piece that the C Suite's got to push is the governance to ensure that, yeah, yes, you're empowered to use it, but there's guardrails and there's sort of security and safety measures and protocols and you know, making sure it's used for the right use cases that it's not built on bias or discriminatory data and those types of things. So that goes back to a governance framework. And I think for most organizations that governance framework should be fairly centralized. So that, that is something that is, everyone understands and is, is bought into and then there permeates to all the applications across the business. But those two things, you know, in my view, the C suite's got to drive the empowerment and the, the governance of it. Okay.

It does seem to make sense if the governance is a whole company wide thing rather than each individual team having to reinvent governance every time because you can have different rules. That's going to get complicated.

Yeah, you'll have teams of legal professionals that you'll need to hire otherwise.

Yeah, I guess no one's had just lots of different lawyers for different AI projects.

Okay.

It does seem like a lot of different teams are going to have a stake in AI. So you're going to start off like the data engineers through to the analysts and all sorts of business and project stakeholders, whatever. So how can you manage all these different interests in AI across your organization?

Yeah, that can be challenging, especially at scale. I think Certain functions within, you know, we just talked about governance. Certain functions within AI can be centralized or should be, should at least be considered to be centralized, I should say, say and that way, you know, you, you have accountability in a, you know, a single, single part of the organization. Now organizations should recognize that, you know, departments or products or however you organize hierarchically. Like they're going to want to be able to own and craft their own AI. They're going to want to be able to use it for the purposes that suit them best. And that's back to that empowerment. They should be empowered to do that because they're going to know the, you know, they're going to know what they need best, you know, better than anyone else and they're going to know how they want to implement it. And so I think for, in terms of, you know, we think of the governance part of it being centralized, the application of it being decentralized at the end of the day all that needs, I think at the end of the day needs to be reported back up to that C suite who's helping to drive it so that they can observe the type of a, it's being used, how well it's being used, what kind of implications is it causing, what outcomes is it driving and then what sort of governance is in place around it so that they can, they can manage it holistically. But you, you know, as you just heard, like, I think a lot of it has to be decentralized in order to create that empowerment while maintaining a central set of concerns around how it's being applied and what sort of guardrails are in place. Okay.

Yeah. I guess the thing is normally when you've got a cross functional team or cross functional project going on, then it's always like it's the other team's fault. But if you've got the C suite watching what's going on, then hopefully the different teams are going to behave and not score ball too much.

Well, especially if it's being reported back. And so the C suite can see, you know, there's a discrepancy between this team and this team and you know, there's a dependency here on the other or whatever, you know, you can detect that and hopefully alleviate it. I think again, if the separations of concerns can be done in such a way that the, you know, the, again the departments or the products or however the organization, but they should be responsible for applying the AI within a construct that is centrally, centrally governed.

Okay, all right, I'd like to switch a bit and Talk about careers. So it does seem like generative AI is having a huge impact on a lot of data careers. So I guess to begin with, how do you think that existing data roles like data analysts, data scientists, how are they going to change due to the rise of generative AI?

Yeah, there's definitely some change in the works. One of the things I'll point to off the bat here is with generative AI, we're talking about moving from structured data to unstructured data. And unstructured data presents a lot of different, like data concerns that are slightly different in terms of their meaning and application with unstructured than with structured. So for instance, when we talk about change, data capture, you know, incremental changes in tabular data, pretty easy to understand when we're talking about changes in data when it's in textual form, how do you manage that? That's a whole new concern. How do you ensure data quality? Again, that's a whole new concern. So I think that will, that will evolve into levels of expertise around working with unstructured data that we haven't seen in the past. The other thing that I would point at is, you know, by most accounts, the amount of unstructured data is at least twice as much, in some cases nine times as large as the amount of structured data that we have. And so organizations have been clamoring for years to try and make the most use of the structured data they have and probably only getting, you know, this much out of it. And now we have this whole pile of unstructured data. So there's a real big opportunity, a real big challenge on the other side of that same coin. But, you know, bringing these worlds together is something unique. I think from a career standpoint. You know, there'll be people that become more experts in working with structured data. You know, if anyone listening here has worked with large language models and worked on prompting, it is super painful. It is a trick into its own trade. You know, the prompts are super sensitive. If you swap out the foundational model, the same prompts don't work the same way. So prompt engineers, like there's already job listings for that. That's become a new thing. I think the role of the data scientist will evolve. I think the data science, there'll still be data scientists who are in the, you know, in the data working, building solutions, certainly. But I think a lot of data scientists will be looked at as more as a higher level concern where they're not concerned about building solutions, but they're concerned about managing and governing Solutions at scale. So that puts them into a role where they're approving other work that's being done by a larger cross section of the organization, ensuring the quality of those models, keeping it to a level of standard that the organization has set. I think the data engineer probably moves into a more of a LLM engineering type of role. Maybe they're prompt engineers, maybe they're the ones that are helping to build out those solutions. And the other role I think probably changes over time is the data steward. You've got this whole new world of unstructured data that they're going to be responsible for from a whole life cycle standpoint. And you know, so their, their job description will be rewritten if it hasn't already.

Okay, that's really interesting that like the sort of consistent theme throughout that is that data science isn't just about numbers anymore. It's really about all these unstructured data types, so text and images and all that sort of stuff. And that actually reminds me, I got asked a question recently that I wasn't quite sure how to answer. So a lot of people, like, traditionally you go, you come to data science, you've got a sort of STEM background, you know, you have science, technology, engineering, maths. And now if everything's going to a natural language interface, you need to be good at sentences as well. So does this mean that if you've got like an English degree background or some other sort of humanities, liberal arts sort of background, does that mean that you, then that's, that's now a viable pathway into data science?

I think so. I have not heard that before, but that makes a lot of sense because we're talking about prose, right? We're talking about how things are worded. And the specificity of that wording has a lot of implication on how the LLM interprets it. And so I could, I could completely see the ubiquitous liberal arts degree that no one thought was any value anymore, being of a, you know, intense value now, particularly because of the way in which these, these models are so sensitive to the way in which you express intent, if you express intent the right way the models work, I mean, it's, they can do a lot of really powerful things, but once you're off by just a little bit or interprets it slightly differently, you know, you're back to square one. So I do actually think there's, there's a role for that to come back into the technology spaces. People that are very well versed on, you know, specific wording, word choice, phraseology, those Types of things so that it communicates well to a AI. Basically, you're not talking to a human anymore, you're talking to AI.

Definitely. Yeah. We've just upended the university system, I suppose. Well, I mean, related to that, I suppose philosophy degrees are now more important as well, with everyone worrying about AI ethics. So, yeah, there's definitely a turnaround there. All right, so we've sidetracked a bit, but you were talking about how some of the existing roles will change. So you mentioned data analyst, data scientists, data stewards. Are there any new roles that you think will come about because of this?

I think there'll be new roles. I think there'll be new roles around developers and how they use large language models. I've already talked a little bit about the prompt engineer. To me, the prompt engineer is a byproduct of this new technology, just as ML Ops engineers were a byproduct of automated machine learning when that came online. However, I don't know that necessarily organizations are going to go out and hire net new talent for that. What I do think is you're going to have a request or a requirement for people in tangential seats to start to move into that. I think that's the most logical means to fill those needs. That being said, there would be certain individuals and programs that start to or already have built into this type of skill set into their programs, and you'll be, you know, you'll be seeing those types of roles being filled by people who have kind of grown up with that as their, their banner to carry. But I don't think you get the scale. So I think there's going to be a lot of transition, as I was talking about, of existing resources that can pick up this new load. Okay.

Yeah. So possibility of like some new roles is probably, like, not completely certain exactly what they're going to be, but we expect some new kinds of jobs going on. So I guess I'd also like to ask about, like, the flip side to this is how can generative AI help people learn these data skills? So, I mean, at Data Camp, for example, we're big fans of everyone being a little bit data literate. So how do you think generative AI can help there?

Well, yeah, it goes back to some of those use cases I refer to as internal. Right. So if you're looking to reskill or upskill or just kind of recraft your own skill set, you know, generative AI is a very efficient way to do that, in my opinion. We're frankly using it internally at qlik to help some of our own initiatives because it's way more efficient to of great create the level of competency in a given area. And so, yeah, I think generative AI has a role to play in that in terms of helping professionals become better equipped to handle this new reality. So yeah, like I said, I encourage those internal use cases because I think there's very little downside to investing in those areas.

And for people who are interested in a career in AI, what skills should they be learning right now?

Because this is all relatively new, especially the generative AI stuff. Right. We're just about a year into when ChatGPT was, was released. So again, I think everyone's playing a little bit of catch up is the right word, but everyone's catching up. So I don't really know that that's the right term to use, but everyone's trying to figure it out. And if, if you're looking for a career in this area, I mean, I would, I would get in and start learning everything you can about large language models, the different techniques around them, the different types of models. What's good about each? What's not so good about each? How do you work with small models versus large models? What is, you know, what is a small model these days? That type of expertise, I think is going to be useful for almost every organization. And so I would start there. That'll bring about a level of competency in an area that very few have it today, which will make yourself valuable, make yourself useful. Now, I think over time there'll be technologies that come online that continue as they always do, lower the barrier, make it easier for the average Joe to come in and get value out of working with a foundational model. So at that point, I don't know that that expertise becomes incrementally as valuable as it is today. However, at that point it'll be more about people who can, that can see an opportunity. They can see how a generative AI solution can transform a particular part of the business and they can connect all those dots to make it happen. That's a skill set that frankly takes. It just takes time to build. It takes time to build experiences. The core set there.

Okay, so there's really like a lot to go on. Like it's some technical skills and then also like understanding how things are joined together. That requires your business skills as well. Yeah, so yeah, a pretty broad skill set. There's probably a lot of different ways into this from different directions, I think.

Yeah, I would agree with that too. Okay. All Right.

Before we wrap up, I'd like to know what are you working on at Qlik that you're excited about at the moment?

Yeah, well, just about everything we talked about. You know, the Qlik is in a unique position, Right. We've got a strong product line around data, strong product line around analytics. And when you think about the application of AI, you need both. You need both the data, and then you need the ability to action anything that comes out of data. So I'm excited by that because we're in a unique position to help our customers in the market in general make use of this technology. The, you know, the intersection of structured and unstructured data, to me, is really interesting, and we're doing some very, I think, innovative and interesting things in there to help try and exploit the opportunities that those two seemingly isolated sets of data can provide. And, you know, foundational models are at the heart of that. You know, generative AI is at the heart of that. So I'm particularly excited about all that stuff because I think it can be transformative, and I think Qlik is in a position that uniquely can help customers avail themselves of that data in a way in which they've never been able to explore before. So I think at the end of the day, we're at the very tip of that iceberg, and I think there's a lot of really interesting things to come from it.

Absolutely. I definitely agree. It's exciting times. Lots more interesting stuff to come. So do you have any final advice for any organizations wanting to adopt AI?

Yeah, I do. I think, you know, we talk a lot about generative AI in this. In this forum here. Generative AI is like the topic du jour and has been for a good period of time here. And that's great because it is transformative. But I think, as I indicated, a lot of people are figuring out how to use it while you have all this other AI that no one's really talking about traditionally that's been around for a long time that works with structured data, supervised learning, predictive modeling. That stuff has immense value, and you can make use of it. Today, you don't have to. Those use cases are better established. You've got a lot of documentation on it. So, you know, one thing I would encourage is, you know, all. All AI matters. That's one of the things that we like to say at Qlik is all AI matters. And to not get like, generative AI has brought up the level of the conversation on AI, but in some cases it's kind of blinded us to the other applications of it. So I would encourage organizations to think about AI holistically, not just with regard to generative. In fact, some people now, I catch them using AI when they really mean generative AI, and I have to remind them, like, there's this whole other class of AI that we know we can use and can drive value with. So, you know, the one message I would leave, given the context of this discussion, is don't forget about the other AI, the traditional AI.

Please remember logistic regression. It's still quite useful. Yeah, I like it.

All right.

Thank you very much, Nick, for your time. That was really informative.

Yeah, I appreciate it, Richie. Thank you.