[DataFramed AI Series #2] How Organizations can Leverage ChatGPT

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI Transformation Journey on datacamp.com when robotic processes were launched in Detroit in the car industry of the United States, a lot of people also then were like, but it's going to displace people. And there was a bit of displacement, but there was also a replacement and an evolution of skills. And I think we're in that similar moment now. As any new innovation comes in, there's going to be things we as humans no longer do. So I think it's a matter of really reframing and redefining foreign.

Welcome to Data Framed in Our Data Framed AI series. I'm Adele, data evangelist and educator at DataCamp, and if you're new here, Data Framed is a weekly podcast in which we explore how individuals and organizations can succeed with data. Today's episode takes a very practical look at things. As with any emerging technology, organizations are now looking at AI and trying to understand how to best implement it and derive value from it. So how do you do that? And more importantly, how do you do that responsibly? Enter Noelle Silver Russell Noelle is the Global AI Solutions and Generative AI and Large Language Model Industry lead at Accenture. She is responsible for enterprise scale industry playbooks for generative AI and large language models and in this role she leads the development of global AI solutions across cloud providers such as Microsoft, Amazon, Google and their ecosystem of partners. She also leads inclusive engineering teams in creating solutions that are industry focused and deliver on the promise of responsible AI at scale. Throughout the episode, we discuss how organizations should be thinking about ChatGPT use cases. We discuss a prioritization framework for enterprise use cases of large language models, common misconceptions about large language models, the technical stack needed to deploy large language model based applications, the privacy angle of generative AI, what AI literacy looks like in the context of these tools, and how to best approach responsible use of AI. If you enjoyed this episode, make sure to subscribe to the show, share it with your friends, colleagues, and make sure to listen to the previous episodes in this series if you haven't yet. And now on to today's episode. Noel Silva Restell, it's great to have you on the show.

Wonderful. I'm so excited to be here. Thanks for having me.

So you are the global AI solutions lead in generative AI and LLM industry lead at Accenture. You've been in the AI and technology space for over two decades now. Maybe to break the ice and to set the stage for our conversation. When was the last time a space and technology data AI was moving as fast as generative AI is moving today?

Yeah, I mean, of course if you look at the data, there probably has not been anything that's moved this fast. However, a lot of the same news, press cycles excitement really did happen as I got involved in AI in production early on in my days at Amazon Alexa. So I feel like a lot of the same things we were saying, this is the first time we put AI in the hands of a consumer. It's the first time consumers can interact with AI freely. All these things we're saying now, so I sense a lot of the same patterns. And actually with we'll probably talk about this today, there's so many things we learned as we built those models, not just at Amazon but also at Microsoft at scale that, gosh, I hope we learn from those mistakes. Like it wasn't too long ago, as you mentioned, 10 years ago. But yeah, so I feel a lot of familiarity to this kind of scaling motion and interest. But I'm glad it's taken even more. Even more people are interested in it now and I think that's an amazing progress.

That's definitely the case. You know, the crux of today's conversation and our special here is how organizations, especially enterprise organizations, can create value with generative AI. Tools like ChatGPT are on top of mind for almost every executive, every business leader. Maybe let's take a step back first and try to understand the different modes of value these tools unlock for organizations. So maybe walk us through your own words. What are the different levers of value creation ChatGPT unlocks for organizations today?

Yeah, that's a great question. I actually try to offer this every time I have a conversation really with anyone about this technology, but specifically those that run organizations. I think there's a couple of different things. One, and probably the first thing that people think about when they think about GPT is customer experience. But it's a little bit different now because in the past we used to try to attain business efficiencies and optimization but almost at the cost of customer experience, almost as a trade off that either you make your business run better or you focus on a better customer experience, but you couldn't have both. And that is something that's really new about the ability for this technology to yes, reduce friction for our customers, make it easier for them to do what they want to do, but at the same time that allows us to deflect work from humans that ultimately had to handle those problems as they went through a flow. And so I think having the ability to do both business optimization and customer experience evolution at the same time with the same model is something that's really, really impactful right now. I think another maybe myth I can bust that I often hear people talk about is that it is GPT or generative AI in general, is a conversational agent. And so it's more about fixing chatbots and making them better. And what we're now seeing is really an evolution on business activity, business to business or process to process communication. I would love, for example, for me to be working in one system and it detects certain signals from my work and be able to trigger natural language requests to other systems so that it's ready by the time I get there. And those are the types of things we're seeing now. Right? How do we get systems to talk to each other without them having to be hard coded to communicate to each other? And that I think is a pretty exciting opportunity as well.

That's a really great framework of thinking about IT systems to systems communication without necessarily hard coding. That you mentioned a myth busting here. Maybe as you're talking to executives based on your conversations, what do you think are the most common misconception other than that myth that you just mentioned, that leaders tend to have about generative AI and ChatGPT and GPT in general?

Yes, I think ChatGPT as it got very, very exciting and a lot more people started using it. I think people tended to think that that was the entirety of the technology that ChatGPT, this web application, which is literally all it is, is a web application, that that was the capability. And what I try to, again, I do a lot of myth busting in my day to day. I try to encourage them to think much bigger than that. That is one web application. But the underlying model is really what's interesting. And that that underlying model, another kind of myth, is that people think that that's the model they have to use. They don't realize that there's a collection of models out there. For example, in an earlier session. AWS has a whole collection of types of models. Different cloud platform providers are going to provide their own collection of foundation models to choose from. So the capability is really quite expansive beyond just what you can do in ChatGPT. And most people are like amazed by what ChatGPT can do. So just imagine how that changes when you can create your own private version of that, when you can fine tune it on your company, your company's data, when you can do it on a cloud platform that you architected to be secure and private. All of these things. We've been working for almost a decade to build this concept of a secure cloud infrastructure to run applications. Generative AI now can sit native inside that environment. And, and most people, when they think of ChatGPT and adding it to their enterprise, they think it's this external thing and they worry about the external nature of it, but really doesn't, in practice it doesn't work that way. We actually are going to internalize it and make it part of our, our cloud infrastructure.

And I really love how you create the separation between ChatGPT the application and GPT the model. We're going to talk about how that fine tuning layer of building APIs on top of your organization see it is going to play out in practice. But first what I want to talk to you about is use cases, right? A lot of organizations right now are looking at this technology and they're thinking, okay, what can I do with this technology? What should I prioritize first? What are the low hanging fruits that I should approach with generative pre trained transformers or generative AI in general? You know, while these models are extremely performant on some tasks, they tend to also hallucinate, they tend to provide wrong answers, they may even provide harmful answers in certain contexts. Maybe walk us through, given that nature of models where they are today as well, what is a use case prioritization framework that you can adopt as organizations are looking to stop these use cases?

Yeah, absolutely. There's so many places to get started and I think one of the important things to do as an organization is to think holistically, to not narrow in on to your point. There are very easily accessible low hanging fruit. But as you think about it, this kind of model is going to be pervasive. It will be disruptive to every line of business, to marketing, to sales, to finance, to legal. So it will be pervasive. Where you start though, I think is relatively, maybe it's in alignment with where most people think they would start, which is in conversations Most of us have invested in chatbots as enterprise companies. Most of us, we're not extremely happy with the results. Right. They're still there though. We still are like, well, it's better than nothing, but it does have room for improvement and in some cases significant improvement. So that's a great place to start. But it's so important when you do choose to start there that you actually think, how does improving that customer experience in the chatbot, how does it change the rest of our workflows? What is impacted because of this bot? Now we can think about document intelligence. Right now we have a GPT model can support document intelligence. So now I don't need to hand off to a live agent. If a birth certificate is needed or insurance form is needed, I can actually do that ingestion, analysis, extraction, all within the conversation and all completely robotic. So I think it's really thinking about more end to end from, I call it front office, middle office, back office. How do we make sure that when we're thinking about this model, we think about all of them? Another easy point maybe to kind of round it out is maybe you don't want to change. Like a lot of people consider their current digital engagement platform as like the lifeblood of their company. Maybe you don't want to mess around with that. Right now with this super innovative technology, there is an incredible opportunity for you to use GPT on the back end to analyze how those conversations are happening, to do named entity recognition, to do summarization. Like we used to have to build one model for each of these things. Now we have one model that does all of these things and that can really help us get efficiencies where maybe we've either they're on our backlog for a data scientist to do, or maybe we've just long forgotten the ability to do it because of our resource constraints. Now you have a model that literally can augment the capabilities of that data science team.

That's a really interesting use case that you mentioned here, which is really the ability to kind of telescope the ability of a data team to create a lot of different natural language use cases, even if they're only on the back end. Right. That create efficiencies. You mentioned here how ChatGPT and GPT models are going to transform every single line of business within the organization. Walk us through, how should leaders think about the human component here? There's a lot of talks about automation, the potential displacement of jobs, what are the actual risks, what's hype versus reality here in this situation? Right. And how do you see that conversation playing out in the future, whether you know you're a leader or you're someone in the organization that's looking at this technology as a potential displacer or threat?

Yes. And I've been talking about this since actually Alexa. Alexa also created this, like, sense that AI was going to replace people. And it's honestly not the first time. Right. When robotic processes were launched in Detroit, in the car industry of the United States, a lot of people also then were like, but it's going to displace people. And there was a bit of displacement, but there was also a replacement and an evolution of skills. And I think we're in that similar moment now. As any new innovation comes in, there's going to be things we as humans no longer do. I often will refer to, like elevators. There was a time when a human was in the elevator and opened and closed the doors for us as humans. And now sometimes you get in and you don't even tell it what to do once you've stepped in the elevator, you've already predefined that beforehand, as scary as that might be. So I think it's a matter of really reframing and redefining. One of the roles I have at Accenture, for example, is to create job aids. So every enterprise needs to really think about how do I want to frame the use of generative AI to augment the ingenuity of the individual. Both Microsoft and even Accenture share this as a core mission in the company, augmenting human possibility. And rather than saying, I'm going to leverage this technology in order to completely replace jobs, what we want to be thinking instead is how do I augment them to do more? It's actually quite capitalist focused, right? Like, how do I not get rid of people, but actually use those people to do more things to accelerate, create a bigger funnel, handle more customers. There is no lack of capability, no lack of growth that a company would want to see. So this actually accelerates growth by creating more efficiencies in your human talent pool. And especially, I think it's interesting when I end up delivering a generative AI solution, the humans that are using it are quite delighted, very happy, as a matter of fact, I delivered a POC not for production, and they're like, I mean, can we just start using it now? Like, they wanted to use it right away. So I think there is another maybe myth there that we think the humans that are going to see this are going to be threatened by it. But I actually find those that I give this technology to are delighted to have it, are excited to use it, are not sad at all to have certain parts of their job that they don't have to do anymore. But I do think the burden's on the leadership of an organization to set that culture number one and to create these job aids that help them use this technology in the right way.

I couldn't agree more, especially on the augmentation aspect. And I think there is definitely an opportunity to reframe the narrative from fear to excitement. I agree with you there. I think Microsoft said when they launched Copilot for Office, the elimination of Drudgery, and I think that message really resonated with me as well. Maybe walk me through when you talk about a organization where large language models are working side by side with people on variety of tasks across different lines of businesses, what is the future of work look like and the needed skills succeed with AI.

So I guess this is probably a great opportunity to maybe introduce to some and reinforce for others the concept of prompt engineering or query engineering. How are we going to because a large language model I'll tell you, I was just using the new the Copilot version for Power BI from Microsoft. So cool. I have the ability to go in and it's a text box just like ChatGPT. I type in what I want and and it generates me a Power BI dashboard. However, Just like with AutoML, if you don't actually know Power BI, you won't even know what to ask for. You won't know what widgets you can use. So there's a level of skillset that is still even though it's going to completely change the amount of time it takes me to build a dashboard, I still have to know what a dashboard is, what widgets display the best information, what my leaders want. Like that's not going to be intuitive to a robotic process. It's not even something we can change because let's face it, our leaders change what they want all the time. Right? So this is where our humans, we're augmented. We don't have to build the models, we don't have to build the model of information ourselves, but we can use this AI to get us to that first draft much, much faster. And I think that's where some of the delight is, honestly is that oh my gosh, I now don't spend half my day building the dashboard, only then to focus on the story the data is telling. I now spend 10% of my day building it. With the augmentation of AI and I spend the rest of my day trying to refine that story. And I think that is where a lot of the human capability right now is lost. People don't even have a chance to think about what they're building. They just spend most of their time, as you mentioned, in the drudgery of generating that content.

It's pretty incredible when you think about the efficiency gains you mentioned here. The Power BI example is an excellent example. At DataCamp we're pretty data science focused as well. One small task that we were trying ChatGPT with is creating a machine learning workflow from A to Z in Python, right? You still need to know to say create a logistic regression, use this accuracy metric, create these features, right? So very, very foundational to have that conceptual knowledge. But just as an aside, it's pretty crazy to imagine a future where creating a dashboard takes 10% of the day, right? Like the time warping effect of generative AI is very interesting. Noel, you mentioned here delivering PoCs to customers and they were very excited about them. I'm not going to ask you about what specific POC look like, but maybe walk us through examples of organizations who have succeeded in adopting large language models with tools like ChatGPT, GPT3, GPT4. Love to hear some of the examples that you've seen.

Of course, now I am a little biased because I'm an mvp, Microsoft MVP in AI. So I have early access to a lot of the technology that we're now seeing come to market today. But I can show you a couple of different organizations that you will eventually be able to see the same things in the products you see every day. That Microsoft just announced Copilot, that it's going to show up in all of our tools. It's not quite there yet, but it's going to be any day now. I always wait because I find like I think GPT launched on Azure Open as an Azure OpenAI service like on a Saturday night, which I didn't know we did that as an industry. So now I'm constantly on alert. But Microsoft is a great example. But also take a look at even GitHub, right? One of Microsoft's of course acquisitions, but it wasn't always. But GitHub ended up building out this Copilot feature. I was a beta tester on that Copilot feature and I also was part of a team that is reimagining what software engineering looks like at a company like Accenture. I mean, Accenture has 700,000 employees, many of them are writing code every day. So what happens when we can refactor, even if it's minutes, 10 minutes, 15 minutes across, hundreds of thousands of employees, that productivity is expansive. So thinking about, how do I use this technology? And again, not just to reduce friction for customer conversation, which is an easy go to when we're using ChatGPT, but how do I ask it questions like you were mentioning asking it functional questions? How do I. I don't know how many of us have spent time looking at code only like the worst thing that happens, you spend an hour, maybe two, looking at a piece of code and someone walks behind you and is like, oh, yeah, yeah, it's right there. And you're like, so one of the things that I have to say, like the pair programming capability of this many companies realize that even if they don't touch their line, even if they don't touch a customer, they can actually just change the way their engineers work on a daily basis. And now, you know, there was a model, Codex. Codex has now been wrapped up into GPT4. So now GPT4 can not only reduce friction for your customers and conversations, it can also help your engineers write better code. We're now seeing this one model do so many vastly different use cases across the enterprise. You can even think about it doing legal triage. Right? Doing RCA for help, desks for employees. I mean, there's this one model now gets to serve a bunch of these capabilities within an organization. And I think that's what a lot of companies are realizing. They start with one, but then they're instantly moving on to what's our second, third, fourth use case we light up across the company.

Interesting. You mentioned legal here. I was reading a report yesterday from Goldman Sachs that said that almost 40% of the legal industry of tasks are subject to augmentation with generative AI. Right. Given how much document triage there is, how much synthesis, understanding, summarization, all of these things. Very interesting. You mentioned use cases here. We talked about successful examples. But I'd be remiss not to talk about the challenges associated with deploying generative AI in production. I want to make sure that we speak deeply about these challenges because this is relatively innocent technology. It's new. There's a lot of risks associated with these technologies for organizations. You don't want to be that brand that says something harmful in your chatbot. Right. So what are the most common challenges that organizations face when deploying these models, and how can they mitigate these challenges?

Yeah, I think the first challenge is really Understanding that you can't just deploy GPT, that it is an enterprise framework, like there's an entire selection or collection of tasks that are required. Everything from. It's very similar. It's almost like MLOps, right? For generative AI, like we still have to do data selection, we still have to create an inclusive data set. Even though the data set in GPT is much or generative AI in general, much smaller, it's still necessary and it still needs to be inclusive. We then also have to consider fine tuning. How do I. And this leads to kind of our questions around accuracy and hallucinations. How do I tell the model what answers I want it to provide to the world? If I am a retail company, do I want someone to come and ask my bot what to do with prisoners of war? Do I want it even able or capable of answering that question? If you go to ChatGPT, it absolutely could answer that. So how do we make sure that we've put guardrails up for what our bot is willing to answer? Because it can. Based on the. I know you had mentioned GPT standing for generative Pre trained Transformer. For those of us that pre training understanding we don't have direct lineage to what that data is, but we do know it's about 499/million websites that were scraped in this process. And all of those websites we know for sure were for the most part written by humans. And those humans are biased. And as humans embellish and they attribute false information and so the model cannot be blamed for amplifying patterns it sees in that data, especially if it's wrong. Oftentimes it's usually modeling a pattern of attribution that if you go and look at the lineage of data, you're like, oh, I see where you might have come to that conclusion, even if it's wrong. So I always tell people models are never wrong, they're just simply untrained. And we do need to take more time, more care and more effort in training them. The two things maybe that I hear the most from industry is one around that concept of hallucination. How do I make sure my model isn't wrong? And that's where fine tuning, that's where prompt engineering, chain of thought prompting, like there's lots of techniques now to control how a response is generated. I love this new role called prompt defender, right. A prompt defend mechanism for like cybersecurity and risks associated with that. I feel like you'll probably talk about that in another session, but there's a lot of opportunity around generative AI to make sure what kind of prompts can be supplied and how do we protect against prompts that are intentionally trying to access data that should not be accessible. All of this extremely important. The second part though, leans more into kind of the responsible AI side of the house, that because we are using a model that is pre trained on all of this data, it will also have systemic biases and patterns of behavior that we do not want to amplify. So you need to have a human in the loop in all of this entire enterprise framework. A human in the loop, meaning a team of humans that are monitoring, right, all this data, monitoring the responses and making sure. Today, luckily, there are companies like Microsoft, like Amazon, that have implemented frameworks, packages, tool sets for monitoring toxicity, for monitoring injustice, for monitoring bias. But the funny thing is those tools are not new. I think what's new, hopefully, is a new type of leadership that knows to look at that data and take action on it in machine learning. We've been looking at bias in machine learning for a long time. But will our leadership, or will the teams that leverage these black boxes, will they know to look at the data that's being provided by these models and say, oh, toxicity, what does that number even mean to us? That, I think is the burden today. How do we get the implementers and users of this technology to think more deeply about what it means to be responsible?

In a lot of ways it seems like we have shifted the gear in the industry. Right. If you kind of want to compare maybe AI to Covid, that's a weird comparison. But you know, Covid happened in early 2020 and there was this shifted mindset that, oh, we're in a global pandemic now. Everything has changed. And I think in a lot of ways generative AI has unlocked that. Oh, we're in an AI era now. We need to adopt era and we need to adopt AI. We need to adopt the best practices and responsible AI technologies, methods, best practices and frameworks. So we do see that gear shift quite a lot and I'm optimistic there. So maybe let's shift gears a bit and talk about the actual data privacy and data management best practices. Because that's very top of mind when think about responsible AI, because there could be risk of data leakage, There could be risk, as you mentioned, prompt injection. Walk us through maybe how organizations should be thinking about data management, data privacy when trying to fine tune APIs on their own data. What are the risks there and what are the best practices that you recommend?

Absolutely. I Think one of the first things so I've been working with through my Microsoft MVP relationship, working with OpenAI for a while, I think a few years ago I had an opportunity to interview some of the executives at OpenAI. Very exciting it was before they were cool like they are now, but very exciting stuff. And the vision was there in the same way. But one of the things that they were discussing and now we see is that investment of Microsoft of a cloud provider. And this is true not just for Microsoft, for aws, for Google. There is a reason why we're going to leverage a cloud provider to support, support our deployment of these types of models. One and maybe we'll talk about it a little bit more. But the tech stack is pretty deep, right? So there's a lot of specialized hardware that's necessary to run a model that's trained on this much data. And not every company needs to roll in new hardware, get a new chipset and do that work. Not everybody's going to want to do that. So what's the next best thing is to run inside of an infrastructure that has been secured, that is already private, that you actually as a company have already spent a lot of money and resources and time and you probably have an Entire team of DevOps people that control this tenant in a cloud provider of your choice. And so those native services already get the benefit of the cloud provider infrastructure that they're running on. So that's why even OpenAI runs, its entire infrastructure runs on Azure, because they themselves as a company realize that that's not. They want to be a model research company. They don't want to worry about elastic scale or elasticsearch or database usage or Blob storage. Like they don't want to worry about, they want to create that, they don't want to build it, so they offload that to a cloud provider. So I think when you're thinking about data privacy and security, I always say it's going to be as secure as you are. Often in these cloud providers it's called a shared response. Which means like the infrastructure, the data center, all of that is going to be controlled by your cloud provider. Then you're going to build your own tenant that you're going to secure. And so that tenancy, that security, that well architected nature of that tenant is going to be completely up to you. But the good news is most of the enterprises today have been working on building secure cloud infrastructure for quite some time. So the benefit now of course is we can use this technology in the context of that so that's.

We've talking here on data privacy and data security when it comes to using APIs. But maybe how they should think about it when the population within the organization is using tools like ChatGPT at the time of recording. I think today or yesterday we had this scandal with Samsung. Engineers were using private data and chatgpt some leakage happened. What should be the guidelines of using these types of models in at work?

Yeah, yeah, this is a great question. So at Accenture, for example, months ago, like in the early part of this year, I was part of. There was just four of us in a room going, okay, we need to figure out how to like lock this down. And so we ended up creating a center of excellence. That center of excellence ended up creating a set of guidelines for the company. We tied those guidelines to corporate policy. We had senior executive leaders of the company disseminate that across the organization and basically say do not use these models in the wild. But the very next thing we had to do, and I think this is the, the responsibility of a company, is that we then also gave them an enterprise sandbox so that they could play. It's very hard to tell someone. I always say when, when you tell someone not to do something, it almost. Maybe I have too many children. You almost are like. I want to say, like don't do it because that's going to make you want to do it more so but in this case it is. We were one very transparent about the risks of exactly what we're seeing in the news today. With some companies there's a risk of that when you share any private data. So sure, go use it if you want to. You can use it to write an invite here or maybe an agenda for a meeting there. But no client data, no Accenture data. But even then there's just risk in just like giving people the leeway to do that. So rather than say no, you can't ever use this ever, like school systems did, we instead said, yeah, of course you can use it. Let's create an enterprise sandbox, which we built on OpenAI with Microsoft. Create an enterprise sandbox and you can now play in the context of our Azure tenant. Accentures secured private Azure tenant, no data ingress or egress out of that tenant that's not already driven by security policies that we wrote that will make the organization feel more comfortable. It'll give people a chance to play. So that's one of the things we do today. We have like this COE in a box where we provide guidance on These policies, we also then provide the ability for you to come and build an enterprise sandbox, whether it's on aws, Azure, Google, right, wherever your cloud provisioning might be.

I love that approach because at one point in time you do preserve privacy. But I think it would be a massive disservice. Especially I can see the education angle, but I don't see the enterprise organization like complete. Barring it. I saw a statistic a while ago that almost 73% of employees would refuse to work for a company that doesn't allow usage of ChatGPT. So I think like ChatGPT is the new remote work in a lot of ways. So yeah, maybe walk me through as well, kind of thinking here about the data privacy component of things. Let's talk about the responsible AI side of things as well. A lot of the times I've been in data science for like six years, seven years now. Transparency, explainability of models have been really important, right? The ability to understand why a model is making the decision it's making. I haven't seen a lot of compelling research on ChatGPT or GPT models that are there, easily explainable, right? Like a random forest is. Have we reached a moment where we have just accepted that black box models are here to stay?

Yes, that's an interesting question. I wouldn't say that. I think there's so many. I mean, I granted, I am in the responsible AI community. I often speak on this topic and I do find that many, like ethicists in general would say, no, we are not at that stage, we are violently opposed to that stage. But I do see companies. For example, one of the really good news about Azure OpenAI service, when it launched, it launched with the responsible AI toolkit built in. And that's new. Like before, it was an option, you could go out and get it, you could apply it, you could add the packages, but it's actually part of the solution. It's a tab in the tool, in the, in the deployment playground. And so that I think is one. Because part of what's different is that there is a call to action by enterprises to say, I don't want to use something, especially a black box AI model that I cannot have some visibility to. I think Stanford Helm, right, came up with this huge battery of metrics to measure not just the accuracy and performance of these models, but also the toxicity and different levels of responsible metrics that we often would measure intentionally if I was doing that on site. So now seeing some. The challenge I think is mostly who's going to play well with the Stanford kind of system of the world. Who's going to make sure that they build. I was just talking to a partner yesterday and they're like, oh, on our roadmap we're building model cards that explain how decisions are made in this neural network. Because as you mentioned it actually we don't want to know, honestly. Like the whole point of a pre trained model is that I don't want to know the details, I don't want to train it myself. But at the same time there has to be some explainability, there has to be the. And auditability. So one of my friends at Meta who runs the marketing organization there, like she was one of those people that when I met her must have said the word audit like 17 times in relation to her data science team in our very short conversation. And I realized that. But auditing a black box model is very, very difficult unless these types of mechanisms are provided. I wish it was a requirement. One thing I have noticed, especially as the economy kind of goes up and down, is that we have seasons of being very interested in responsible AI and doing the right thing and measuring and checking for accuracy. But as soon as things get tight, as soon as resources become less available, the economy takes a turn. As soon as that starts to happen, the teams that often get shed in those times are these responsible AI teams. Right. And so that is something I'm looking forward to as we move into like maybe a conversation on regulation or policy that we see where this isn't an option. Right. Like building responsible tech is not the burden is not on the CEO. Right. Like I always tell people I'm one acquisition away from the death star in any solution that I build. Like, I never know who's going to own it, who's going to buy it, so how do I protect it? And honestly, I feel like regulation is one of the shining lights at the end of the tunnel that I'm looking for to help make this a requirement and not an option for enterprises.

So let's jump in in regulation here. Walk me through. What do you think good government regulation for AI and these types of tools should look like?

Yeah, I probably would start with the word collaboration. Right. I think there's very and not just collaboration with the big tech companies. I'm always surprised at how many lobbyists are employed by the big tech companies in these conversations, but really about consumers creating what I call kind of an inclusive data set. Right. As we build these, I think there are lots of good examples of this. The EU just came out with who knows if it'll pass? But, and maybe by the time someone hears this, maybe it will have. But even if it doesn't, it's a great set of guidelines for organizations to think about what to talk about. And the way that they built that was again, by building an inclusive group of people across industries. Because when you build regulation, you really, I always think, let's go for regulated industries first. Usually they're the highest risk. They also know regulation. So they're not foreign to the idea of being told what to do. Let's face it, engineers, data scientists, technology, we're not used to being told what to do. It's going to be hard culturally to shift that, that monolith. However, organizations that are already this way, finance, healthcare, life sciences, there's opportunities for us to put in guardrails that are not far from guardrails they already have in other parts of their organization that will refine and protect users in some of the most important situations. Right. Where their finances and independence is concerned or where their healthcare is concerned. So that's some of my guidance. And I definitely have been working with policymakers. I recently spoke at the consulate, the Canadian consulate in New York, to talk about where should we start. It's almost the same conversation you mentioned earlier with enterprises like what do we start with? Because if we try to boil the ocean with regulation, we'll just never, I mean, we're there now, we've been doing this for 20 years and there's still no regulation. So how do we pick something that we can identify and go after and begin without trying to do so much that we never get started?

Yeah. And maybe harping on what was, you know, a big story in the generative AI space a few weeks ago, that Future of Life letter on the moratorium of stopping AI research. I see this debate happening within the AI community and data science community is that regulation should be on the application level, not on the research level. Where do you stand on that debate?

Yeah, I would definitely lean more towards like research. Research and academia in general are also under a different level of regulation, a different level of usage. They are not production based, they're not making money, which drives weird behavior, let's face it, in corporate organizations. So, yeah, I do believe that research and it goes along the lines of what I mentioned before. If I tell people not to do something, it's just going to fuel the fire for those very specific people who are like, yeah, we're going to do this. I don't think there's harm in Research, it actually informs us as to things. Like there's lots of things that we researched. I'll give you an example at Amazon Alexa, we in our research organization found out that we could actually answer a question that was being asked to the device within about three words of the sentence meaning far. But like, even if the three words were not contextually specifically about a topic, we could tell what question you were answering with 98% accuracy. However, it's very weird to have a device answer a question that you only have in your mind. Right? Like it creeps people out, like people don't like it.

So what I want to end on here in our conversation is we talked about, in our discussion about responsible AI, how organizations should adapt. But let's talk about the culture maybe a bit more deeply. We talk about data literacy quite a lot here on Data Framed. And I think AI literacy is something that we need to think about as well. We need to kind of combine data and AI literacy. Especially, as you mentioned, quite a lot of folks within organizations are interacting with tools like ChatGPT. How do you define AI literacy in the age of generative AI?

Yeah, so I often, I actually have a way that I frame this myself. I started a company called the AI Leadership Institute around this exact kind of framework. And I always think, and I even explain myself when I introduce myself. I often will say this, that I like to go from the boardroom to the whiteboard to the keyboard. And I actually say any technologist should be able to do this, right? That as we build things, especially artificial intelligence, especially models that will serve as a foundation model to other companies, that we want to be able to explain it to a business from a business perspective, business value, business interests, but then also have a pretty technical discussion on architecturally, what does this take? One of my biggest concerns is around sustainability, like how much resources does this consume? What, what is the capacity that is required? Thinking about going from like that business level view to a highly. It's not high level, it's just an architectural level view and then putting your fingers on the keyboard and building something. So I will go into an organization and do an executive briefing to get everyone in the company familiar with. And as you mentioned, it's not just about artificial intelligence or even these foundation models. It's about what it means to be a data driven organization, right? What it means to use data to build and leverage a model like GPT in order to gain insights. I love GPT models that I can ask questions like what does my pipeline look like this week? What are the three accounts I should go after. Those insights are only available if we gathered the right data, trained the right model, and informed it to give me that information at the right time with the right levels of security. So I think it's really important to have that in an organization. End to end understanding. And then of course, the keyboard part I really like. I encourage organizations, once the executives and the board have the same understanding, senior technical leadership is all on board. The best thing to do is get business and technology into a room and call it a hackathon, call it a innovation session. But build something and you'd be amazed. I've done this at the Metropolitan Museum of Art. I've done it at Abbey Road Studios. You'd be amazed at what ends up happening when people who have the problem, the business, talk to people with the tech who have a solution. Like, we came up with some pretty incredible stuff. One of them was even, I think we had a story on NBC Nightly News because of its such interesting and most importantly, accessible use cases. But without that, as you probably know, as data scientists, we think kind of in our own little bubble. And so it's really important for us to connect with the business and use our skills to make their vision a reality.

In a lot of ways, marrying that skill set, but also having the different folks within the organization speak to each other is an important aspect of creating that culture of AI literacy and data literacy. But maybe as you look at the future of work and how leaders can foster AI literacy, do you think that the current education system is well prepared for the changes that are coming within the job market, the skill sets needed? What do you think needs to change to bridge that gap?

If not, I would say that there are pockets of educational systems with leaders that are proactively technology focused and technology leaning. I will tell you, in Florida, we have a mayor in Miami who's very technology focused. Mayor Suarez, like technology first. And as a result, the schools, universities, colleges in this area are also kind of building on that momentum. And then. So I recently had a fireside chat with the president of Miami Dade College. I'm in Miami, Florida. And she was one of the first to say, I will not ban this technology. I will equip my teachers and my professors and my students on how to use this to solve problems. And I thought, like, that's, that's kind of what we need. But at the same time, there's a bit of evangelism that's necessary to go spread that beyond just those who are lucky enough to have leadership that drives that conversation. What happens if you're in an organization, in a university, in a school that doesn't see things that way? How can you show them the light? And so I often say, build something. So one thing I did for Miami Dade, I built what, what I eventually called the Intelligent Student Assistant. But it literally leveraged a GPT or a large language model. Wasn't GPT at the time a large language model to reduce the amount of workload a faculty member got on scheduling TAs. And people would constantly ask what their grades were, when grades were coming out, did they get an assignment? All things that we could check within a system. And so that Intelligent Assist ended up offloading 30% of the time of a faculty member. And like I said, the faculty was not worried that their jobs would be taken. They were ecstatic that they did not have to do this work anymore. And we did that without anyone asking. We just built it to be helpful. And it lit up the entire network of colleges to be like, what can we use this for? So I think that's why the power of learning by doing is so powerful. Because if you build something useful, people will then start to see the opportunity that lies within this technology.

And you mentioned here the importance of evangelism. I think you're doing an excellent job of that, Noel. As we close out our chat, I opened up our conversation saying, when was the last time you saw space and technology AI move this fast? I want to end my conversation on a similar note. Where do you see the space being in 12 months from now?

So I think interestingly enough, we've already started to see it, but the models will just get better and there'll be more provisioned models specific to industry. We're already starting to see it, right? I think Bloomberg recently released or at least announced its Bloomberg meaning financially specific GPT model. We'll see Microsoft Research has one on biogbt. There's lots of. Right now it's in its nascency. But what happens to all these companies that are serving medical communities, patient communities, and now they don't even have to fine tune the model that much right now. That's still a gap that I still have to go into a company and I still have to get it data and fine tune the model to get it to answer what happens when we reduce what I call the time to value or time to market for these companies? So I feel like in 12 years we'll start to see an even bigger shift of organizations of all sizes starting to leverage this technology in meaningful ways.

That is going to be very interesting. I'm excited for the App Store for models where I'm going to be able to download my legal GPT, my biogpt, all of these things. Yeah, you should pack that.

Yeah.

I mean, Hugging Face is already doing.

Yeah.

Yeah. Noel, before we wrap up, any final call to action before we wrap up today?

Yeah. I would encourage everyone to really learn by doing. There are some incredible GitHub repos available if you are a practitioner. Even if you're not. I always reference GitHub as kind of tracing paper that artists use. Right. Even if you're not proficient, you can use tracing paper and create pretty fantastic artwork. It's kind of like paint by numbers. A lot of our GitHub repos are this way. Microsoft, AWS, Google have all released samples, generative AI samples that you can build. Some of them even have released all of the infrastructure as code. I will not name it per each one, but you know what I mean. The scripts that are necessary, so you hit one button, launch a couple bash or shell scripts, and it will build for you in your infrastructure. Just get in there and start playing around with this. This is exactly the time. Don't be that person two years from now. That's like, I remember hearing about that. I should have dove in. Just dive in now. There's no harm in that. And learning by doing will set the path for your way forward using this technology.

I couldn't agree more. Learn by doing is something that we live by here at Data Camp. Thank you so much, Noel, for joining us today. It was a really insightful discussion.

Yeah. Super fun. Thank you for having me.