The Challenge of Enterprise AI Adoption

Today on the AI Daily Brief the Challenge of Enterprise AI Adoption the AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. To join the conversation, follow the Discord link in our show. Notes. Welcome back to the AI Daily Brief. I am once again on the road this week and this time I am headed to speak about enterprise AI adoption for a big company in the Midwest. And so I thought what would be interesting is to read the most recent piece from Professor Ethan Moloch on his One Useful Thing blog called AI in Organizations. Ethan is getting at some of the very same issues that I will be dealing with in my presentation. And so what we're going to do is we're going to first read Ethan's piece and then we'll come back and do a little bit of discussion. Ethan writes, over the past few months we've gotten increasingly clear evidence of two key points about AI at work. 1. A large percentage of people are using AI at work. We know this is happening in the eu, where a representative study of knowledge workers in Denmark from January found that 65% of marketers, 64% of journalists, 30% of lawyers, among others, had used AI at work. We also know it from a new study of American workers in August where a third of workers had used generative AI at work in the last week. ChatGPT is by far the most used tool in that study, followed by Google's Gemini. We know that individuals are seeing productivity gains at work for some important tasks. You have almost certainly seen me reference our work showing consultants completed 18 different tasks 25% more quickly using GPT4. But another new study of actual deployments of the original GitHub copilot for coding found a 26% improvement in productivity and this used the now obsolete GPT 3.5 and is far less advanced than current coding tools. This aligns with self reported data. For example, the Denmark study found that users thought that AI halved their working time for 41% of the tasks they do at work. Yet when I talk to leaders and managers about AI use in their company, they often say they see little AI use and few productivity gains outside of narrow permitted use cases. So how do we reconcile these two experiences with the points above? The answer is that AI use that boosts individual performance does not always translate to boosting organizational performance for a variety of reasons. To get organizational gains requires R and D into AI use and you are largely going to have to do the R and D yourself. I want to repeat that you are largely going to have to do the R and D yourself. For decades, companies have outsourced their organizational innovation to consultants or enterprise software vendors who develop generalized approaches based on what they see across many organizations. That won't work here, at least for a while. Nobody has special information about how to best use AI at your company or a playbook for how to integrate it into your organization. Even the major AI companies release models without knowing how they can best be used, discovering use cases that they are shared on Twitter. They especially don't know your industry, organization or context. We are all figuring this out together. If you want to gain an advantage, you are going to have to figure it out faster. So how do you do R and D on ways of using AI? You turn to the crowd or the lab? Probably both tactics for the crowd One of my advisors during my PhD at MIT was Professor Eric von Hippel, who famously developed the concept of user innovation that many key breakthrough innovations come not from central R and D labs, but from people actually using products and tinkering with them to solve their own problems. A key reason for this is that experimentation is hard and expensive for outsiders trying to develop new products and but very cheap for workers doing their own tasks. As users are very motivated to make their own jobs easier with technology, they find ways to do so. The user advantage is especially big in experimenting with generative AI because the systems are unreliable and have a jagged frontier of capability. Experts can easily access when an AI is useful for their work through trial and error, but an outsider often cannot. From the surveys and many conversations, I know that people are experimenting with AI and finding it very useful. But they aren't sharing their results with their employers. Instead, almost every organization is completely infiltrated with secret cyborgs. People using AI at work but not telling you about it. Here are a bunch of common reasons people don't share their AI experiments inside organizations they received a scary talk about how improper AI use might be punished. Maybe the talk was vague on what improper use was. Maybe they don't even want to ask. They don't want to be punished, so they hide their use. They are being treated as heroes at work for their sensitive emails and rapid coding ability. They suspect if they tell anyone it is AI, people will respect them less, so they hide their use. They know that companies see productivity gains as an opportunity for cost cutting. They suspect that they or their colleagues will be fired if the company realizes that AI does some part of their job, so they hide their use. They suspect that if they reveal their AI use, even if they aren't punished, they won't be rewarded. They aren't going to give away what they know for free, so they hide their use. They know that even if companies don't cut costs and reward their use, any productivity gains will just become an expectation that more work will get done. So they hide their use. They are incentivized to show people their approaches, but they have no way of sharing how they use AI, so they hide their use. So how can companies solve this problem by taking these things seriously? First, you need to reduce the fear. Instead of vague talks on AI ethics or terrifying blanket policies, provide clear areas where experimentation of any kind is permitted and be biased towards allowing people to use AI where it is ethically and legally possible. As a side note, many internal legal departments have an outdated view on the risks of AI. Rules and ethical standards are obviously important but need to be clear and well understood, not draconian. And fixing policies isn't enough Figure out how you will guarantee to your workers that revealing their productivity gains will not lead to layoffs. Because it is often a bad idea to use technological gains to fire workers at a moment of massive change. For companies with good cultures, this will be easier, but for those where employees have little faith in management, you may need to resort to extreme measures to show that this time you aren't going to use technology as an excuse to lay off workers. Psychological safety is often the key to a willingness to share innovation. Second, you need to align your reward systems. Figure out how to reward people for revealing AI use. If productivity gains happen, workers need to benefit as well. That might mean giving really big rewards for really big gains. Think cash prizes that cover months of salary, promotions, corner offices, the ability to work from home forever with the potential productivity gains possible due to LLMs. These are small prices to pay for truly breakthrough innovation, and large incentives also show that the organization is serious. Third, model positive use. Executives should be obviously using AI and sharing their use cases with the company. Barry Erdos, for example, CEO of JP Morgan's Asset and Wealth Management Group, talks about how the firm is prioritizing AI use at the leadership level and incorporating their AI experiences into their strategic thinking. Once they become users, managers can encourage their employees to turn to AI first to try to solve problems. For example, Cynthia Gumbert, CMO of Smart Bear, told me that when teams come to her for resources for a new project, if she thinks AI could help, she tells them, prove to me you can't do it in AI first. Then maybe I will fund the work. Give others the opportunity to show their uses as well. Public facing events like hackathons, especially including non technical experts and prop sharing sessions often work well. You also need to think about how to build a community. AI talent can be anywhere in your organization. How are you finding the people who are enthusiastic and talented and helping them share what they have learned? Of course you also need to give your employees access to tools and training for tools. This usually means giving them the ability to play directly with a frontier model, probably Claude 3.5 GPT 4.0 or Gemini 1.5 and systems like OpenAI's GPTs, Claude's projects or Google's gems that allow them to develop and share more complete solutions. Training is a bit more of a challenge because there's still a lot of discussion over ways to use AI, but even just an introductory session could give people permission to innovate the innovation Talent for AI is inside your organization. You need to create the opportunity for it to flourish. The Crowd can help, but there's also a role for a more focused innovation effort. The Lab Tactics for the Lab as important as decentralized innovation is, there is also a role for more centralized effort to figure out how to use R and D in your organization. The lab needs to consist of subject matter experts and a mix of technologists and non technologists. Fortunately, the crowd provides your researchers. Those enthusiasts who figure out how to use AI and proudly share it with a company are some of the talents you will use to staff the lab. Their job will be completely or mostly about AI. You need them to focus on building, not an analysis or abstract strategy. Here is what they will build. Build AI benchmarks for your organization. I've ranted about the state of benchmarks in AI before, but almost all of the AI labs test on coding and multiple choice tests of knowledge. They don't tell you which AI is the most stylish writer or can handle financial data or best read through a legal document. You need to develop your own benchmarks. How good are each of the models at the tasks you actually do inside of your company? A set of clear business critical tasks and criteria for evaluating them is key. Without these benchmarks you are flying blind. You have no idea how good AI systems are and even more importantly, you do not know how good they are getting. If you had benchmarks you would know whether new O models represent an opportunity or threat or if they are closing the gap with human performance. Most organizations have no idea. Next, build prompts and tools that work. Take the ideas from the crowd and turn them into fast and dirty products. Iterate and Test them, then release them into your organization and measure what happens next. Build stuff that doesn't work yet what would it look like if you used AI agents to do all the work for key business processes? Build it and see where it fails. Then when a new model comes out, plug it into what you built and see if it is any better. If the rate of advancement continues, this gives you the opportunity to get a first glance at where things are heading and to actually have a deployable prototype. At the first moment, AI models improve past critical thresholds finally build provocations and magic Many people have failed to engage with AI yet if you are following AI closely, there is a good chance you see something amazing or disturbing on a regular basis. Demos and experiences that get people to viscerally understand why AI might change or alter your organization and a value all their own. Show how far you can get with an impossible task with AI or what the latest tools can accomplish. Get the people going. The crowd innovates and the lab builds and tests. A successful internal R and D effort likely involves both. This is just a start. In the longer term, innovation is not enough to thrive if AI abilities continue to advance. Instead, companies will need AI aware leadership. Our organizations are built around the limitations and benefits of human intelligence, the only form we've had available to us. Now we must figure out how to reconfigure processes and organizational structures that have been developed over decades to take into account the weird intelligence of AIs. This requires going beyond R and D to consider organizational structures and goals and what the role of people and machines are in the organization of the future. The right way to do this is not yet clear, but should be something companies and the consultants and academics who advise them need to start working on now. And yet this may not be radical enough. The explicit goal of the AI labs is to build AIs that are better than humans. At every intellectual task. They have promised that soon we will have agents, autonomous agents with goals that can plan and act on their own. Ultimately, as OpenAI's roadmap shows, they believe they can create AIs that can do the work of organizations. None of this may happen, but even if just some of it does, the changes to organizations become far more profound in ways that are difficult to imagine today. For companies, the best way to navigate this uncertainty is to take back some agency and begin to explore this new world for themselves. Today's episode is brought to you by Fractional when we wanted to build an AI powered feature of Superintelligent, our AI tool Finder, I went straight to Fractional. Fractional Team is a group of senior engineers in San Francisco working on some of the most exciting projects in applied AI. Working with them is basically like hiring an absolute top flight AI engineering team, but in a way that you can customize exactly for your particular needs. Like I said, that AI tool Finder feature that we built with them is already a key part of the super intelligent platform and we are working on something new as well as Fractional works with everyone from startups to the Fortune 500 and to request a free consultation you can go to Fractional AI. If you want help identifying and building AI projects for your business, then I highly recommend you hit pause, open a web browser and go to Fractional AI to request a free consultation. Today's episode is brought to you by Venice. The leading AI companies store your entire conversation history and attach it to your identity forever. That's every question you ask, every answer you receive, every image you generate, every thought you share with the machine. It's all being spied on. If you trust all the company's hackers and NSA board members that will ever have access to your AI conversations, then rejoice for you are well served. For the rest of us, Venice is an alternative. Venice is a powerful AI app for text, image and code generation that respects you as a sovereign individual and believes privacy and free speech are not only human rights but necessary for civilizational advancement. Private, permissionless and uncensored, you can try it for free without an account. AIA Daily Brief listeners receive a 20% discount on Venice Pro. Visit Venice AI NLW and enter the discount code NLWDailyBrief. That's NLW Daily Brief, all one word. Today's episode is brought to you by Superintelligent. Every single business workflow and function is being remade and reimagined with artificial intelligence. There is a huge challenge, however, of going from the potential of AI to actually capturing that value. And that gap is what superintelligent is dedicated to filling. Superintelligent accelerates AI adoption and engagement to help teams actually use AI to increase productivity and drive business value. An interactive AI use case registry gives your company full visibility into how people are using artificial intelligence right now. Pair that with capabilities building content in the form of tutorials, learning paths, and a use case library, and superintelligent helps people inside your company show how they're getting value out of AI while providing resources for people to put that inspiration into action. The next three teams that sign up with 100 or more seats are Going to get free embedded consulting. That's a process by which our superintelligent team sits with your organization, figures out the specific use cases that matter most to you, and helps actually ensure support for adoption of those use cases to drive real value. Go to BeSuper AI to learn more about this AI enablement network. And now back to the show. All right, so another great one from Ethan. This is where his interests and my interests intersect most directly. So I was really excited to see this piece. So basically what Ethan is trying to do here is give companies the start of a blueprint that, if not specific, at least give some ideas of where they can start. One of the things that rings most true is the long list of reasons that people are keeping their AIU secret. The good news is that many of these are based in organizational culture that can be changed. One of the biggest starting points for companies is just being clear that they actually want people sharing what they're doing with AI. Now, that doesn't solve concerns like people believing that AI is just going to be cost cutting, so they want to keep it a secret. But even just the starting point of an organization standing up and saying, we want to hear how you're doing things makes a big difference. Ethan's last bullet in this list where employees are incentivized to show people their approaches but have no way of sharing how they use AI is the exact problem that we're trying to solve with Superintelligent. What we found over the last year of digging into the AI enablement problem is that there is a critical information failure around the visibility of AI usage inside organizations. This is basically the AI equivalent of if a tree falls in a forest and no one's around to hear it, does it even make a sound? If an employee figures out an AI use case that improves their business workflows but they don't share it with anyone else, does it actually change the organization? The invisibility of AI usage is having impacts on two levels. On the one hand, it's stopping effective employee adoption. Basically, we're creating a scenario where every individual employee has to be their own little creative tinkerer and experimenter, doing all these tests on their own, trying to figure out what works, and hopefully stumbling into something that actually makes their life easier. Because people don't have a mechanism to share what they're learning. It means that mistakes get repeated over and over again and that the things that are actually working don't have any mechanism to scale. Even more broadly, if you look at past technology adoption, the Pattern is always that a very, very small percentage of people, a tiny handful of users, basically does all that tinkering for everyone else, figures out the use cases and then everyone else just copies them. It is basically completely ahistorical from the way that technology adoption usually works to force everyone to be their own tinkerer and use case creator. Most people just aren't going to be in a position to actually figure that stuff out. Okay, so the visibility problem impacts on the one end employees who aren't adopting AI as effectively or as fully as they could. But this invisibility of AI usage also has big impacts on the organization level. Without actually understanding how people are using AI, people who are in leadership positions don't have good information with which to make strategic decisions about AI. And by that I mean both AI strategy itself, like which tools to adopt, but also which areas of business strategy might be impacted by AI as well as the net of this is that most organizations find themselves somewhere between not knowing exactly where to start and Proof of Concept or Pilot Hell, an incredibly small number of organizations have built systems that can actually scale use cases that work across the organization. And yet at the same time, there is such a strong, clear sense that these tools are going to impact every business process that just giving up isn't an option. And so organizations find themselves stuck between, on the one hand, the pilot in Proof of Concept Hell, the inefficient adoption, the endless use case experimentation that happens in individual silos and can't be scaled, and on the other hand, the pressure that they must figure out AI adoption by hook or by crook. Now, there are a lot of other challenges as well. One of the big ones is that although all of these AI companies are looking to the enterprise for their business model, at the same time, successful enterprise adoption is not going to be what dictates their success or not. What dictates the success of these AI companies in this race right now is model innovation. And so not only is there not a huge appetite to support AI adoption with AI enablement systems inside the enterprise, anything that takes away time from model innovation is actually potentially a reason that a company might not win. This is just to put an exclamation point on Ethan's argument that organizations really are going to have to figure this stuff out for themselves. The good news is that while Ethan is right that this is going to require a ton of individual customized effort, enterprise by enterprise, organization by organization, there are increasingly more blueprints, the beginnings of communities of practice, software like Superintelligent, that's trying to solve the AI enablement issue. There is a new infrastructure that is being built currently to support organizations. It's just only coming online right now. These are the challenges that we think about here every single day. It's what where I spend most of my time. So if this is interesting to you, if this is something your organization is trying to solve or something that you're trying to solve for organizations, please reach out. I'd love to hear about what you're doing. Of course, check out Super Intelligent, which you can find@BESuper AI. But for now, big thank you to Ethan for another great thought provoking essay. And until next time, peace. Sa.