#223 [Radar Recap] Charting the Path: What the Future Holds for Generative AI

Oh, hello everyone and welcome to our penultimate session of the day at Data Camp Radar. I am so excited for this one on what the future holds for generative AI. Everyone, do give us a lot of love in the emojis. We want to see it. Let us know where you're joining from in the chat. Everyone give Thomas, Ido, Nick a lot of love today because they're going to be sharing a lot of excellent insights. I've already been enjoying the behind the scenes conversation quite a lot, so we're in for a treat today. You know, there are various ways by which we can look at the generative AI ecosystem today and its future. You know, from how the technical landscape will evolve to how LLMs themselves will improve, to the infrastructure to the adoption, the future of work. And you know, how generative AI in general will really change society and the tech space as a whole in the future. So luckily today's guests all come at the generative AI. You know, come from the generative AI ecosystem, but from different angles. And it will be awesome to pick their brains and see how they view the future of AI. So first, let me introduce Ido Liberty. Ido Liberty is the founder and CEO of Pinecone, the managed vector database that helps build knowledgeable AI systems. He is also the former head of Amazon AI Labs and the research director at Yahoo. He Ido is key in building Amazon SageMaker. If we have any fans of Amazon SageMaker, let us know in the chat. And has computed to over 100 academic papers propelling AI research forward. His company Pinecone was founded in 2019 and is now valued at $750 million. That's not bad at all. Ido, it's great to see you.

It's great to see you.

I wouldn't if that valuation is right, but other than that, it was.

And next we have Thomas Tungus. He is the general partner at Theory Venture. Thomas has been a venture capitalist for more than 15 years. Before founding Theory, he has worked with companies including Looker, Monte Carlo, Mother Duck, Omni Hex, Arbitrum and Mistin Labs. Quite a few well known names in the data and AI space. And he writes@tomtungush.com which I highly recommend that you check out his blog. One of my favorite reads of the week. Tom, it's great to see you.

Pleasure to be here. Thanks for having me on.

Awesome. And last but not least is Nick Alprin, CEO of Domino Data Lab as co founder and CEO. Seems like we lost Nick, so I think we're having some technical Issues Nick will be again. No worries. Yeah. So Nick is the CEO of Domino Data Lab. As the co founder and CEO of Domino, Nick had a front row seat to the transformative impact that Domino's platform has unleashed for its customers. Nick started Domino in 2014 after spending nearly a decade building platforms for data scientists at Bridgewater, the world's largest hedge fund. Since then, Nick has built Domino into an industry leader whose ongoing innovation has continuously reshaped the standard for data science and AI platforms. He holds a BA and a Master's in Computer Science from Harvard. Nick, it's great to have you on.

Nice to be here, thanks.

So, just a few housekeeping notes. There will be time for Q and A at the end, so make sure to use the Q and A feature before asking before asking any questions. If you have any other messages or want to engage in the conversation, use the chat feature. If you want to network, add your fellow attendees on LinkedIn. We highly recommend joining our LinkedIn group. Any LinkedIn posts will be auto removed immediately just to make sure that the chat is high signal to noise ratio. So let's talk about the state of generative AI today. You know, we've been talking about this behind the scenes, but I'm very excited to pick your thoughts here. You know, all of you approach generative AI from different angles. You know, Thomas, you're investing in the space. Either you work at Pinecon, you're literally building the infrastructure that is powering a lot of generative AI use cases today. And Nick, you're building tooling that makes building and managing AI models much easier. So maybe let's set the stage for this conversation. How do you view the current state of generative AI today? Tom, I'll start with you.

Well, I think we're at a really exciting time. You look at the model, performance is rapidly improving. Where small, medium and large language models are, there's a score called the mmlu, which is high school equivalency. And all those models are performing pretty well. The Llama model, you have open source and closed source that are both thriving with OpenAI and anthropic being closed, and then Mistral and then the Llama family of models being open source. There's a lot of innovation at the infrastructure layer. You know, even the chips, right? We're seeing pretty meaningful improvements. Nvidia, with the H200, you have AMD launching chips that are 30% more powerful than the current H1 hundreds and then even startups that have architectures that profess to be 10 times faster. So there's a lot of innovation at the infrastructure layer and I think as that evolves and improves, there's a parallel desire from businesses to really take advantage of this. There's a dominant feeling that this is a big wave for good reason. And so we see a lot of innovation, a lot of desire like enterprise search is a major use case. Asking questions about a knowledge base, customer support is a major use case. Applications in legal accounting, code completion, those are probably the five most active areas of investment today. Then in terms of the future, where we're going is making sure that these models are accurate over multi step processes. And once we're able to unlock that, then I think you'll actually see some massive productivity improvements because we will be able to delegate tasks that take a lot of time and have a computer do it instead of us.

Okay, that's great. And Ido, you're building infrastructure to power a lot of generative AI tools. Let us know how you see the state of turn to VA today.

Yeah, I shared Thomas's Tamashit. I don't know, I'm not sure how I'm pronouncing your name in his in the right way, am I?

You are. It's an sh at the end. It's kind of you to ask. Thanks.

The. Yeah, I mean the space of course is rapidly evolving and so on. I think we see a few different trends. I think the biggest one is enterprise adoption which sort of like started in, you know, fits and hiccups last year. And I think this year people really started to invest deeply in that. And for that they, their models and their AI in general to be knowledgeable and trustworthy and dependable and governed and like they sort of like the excitement of the new technology kind of hits the, the rocks of reality of what it means to be able to actually work in, in an enterprise. I think we see a lot of that and at the same time, I mean I share the, the excitement. I mean there's, there's a ton of improvements. I think agents and assistants are going to be very big. We're going to improve a lot of them with more knowledge from vector basis and other improved models and better mechanisms and so on. So yeah, there's a ton coming and there's a lot of energy and desire to build on those tools. So.

Mm. So we're definitely gonna unpack a lot of that. And maybe Nick, from your perspective, walk us through how you view the state of generative AI.

Yeah, well, maybe I'll be a little more contrarian or A little less sanguine than Tomash and Ido. You know, I like the technology. Innovation is very exciting from my perspective, working with customers and most of our customers are large, large global enterprises. You know, I'd say we're, my take is we're still kind of in the inflated expectations period of the hype cycle. And, and what I mean by that is there's a lot of prototyping, there's a lot of experimentation, but most of the real production use case, not all of them, but most of them we're seeing are still sort of internally facing like Tomas said, you know, enterprise search, question and answer, chatbots and look like that's fine, but I don't, you know, that stuff is not world changing. And where I've seen companies really get excited about initial ideas for products based on Genii that would change a business model that would really be customer facing. They have exciting prototypes but they can't get across the finish line because there's still too many issues around accuracy around, around, around safety or around roi, frankly. And so you know, I think, look, I think that maybe we're jumping, jumping a little ahead. But, but I, I think there's a huge opportunity for Gen AI to impact services, heavy businesses, consulting businesses, graphic design agencies. But, but I think that for a lot of other businesses, I worry actually that the infatuation with Gen AI has distracted business leaders from the benefits and the potential of traditional predictive machine learning, traditional AI. And I think that in some ways has sort of set us back on the potential for all AI to really transform businesses and have positive impact on the world. So that, yeah, like I said, maybe, maybe a bit more provocative take, but that's how I see things right now.

Yeah, maybe Tom and Ido react to that. So how do you view kind of the adoption landscape today? So I'll let maybe Ido first. I'll let you take that.

Sure.

I agree 100% that there is a ton of experimentation and most of the energy in this field is spent with understanding how to run these things, how to make your applications first of all perform what you want them to perform, be governed, be dependable, not make silly and embarrassing mistakes and so on. And that's hard. And that's hard. And I agree with Nick 100% that some people might have unrealistic expectations from AI.

That said.

I do think that a very large set of new kinds of workloads is now possible that was not possible with traditional machine learning and, or whatever I would call them like predictive models. Whose job is mainly to score and predict and rank and do stuff that we do very well today already, and I agree with Nick 100%, are incredibly valuable and people should invest in them. But the ability to consume language and generate language in a meaningful way is incredibly powerful. And I think, yes, a lot of people don't know how to commercialize it yet within their business, and they're experimenting, but I think a lot of them already know what they want to do and already know that this is the answer. Now the question is how to build it and how to get there. And, you know, when can we launch this?

Okay, Tom, I'll let you react as well.

Yeah, I agree. I think we're still early, really. People are still trying to understand exactly what's going on, what the technologies are good at, what they're not good at. If there are security concerns around data loss, which. And there's this dynamic where historically the chief Information Security officer, the ciso, has been the one that's been responsible for securing these companies. And now the heads of data, heads of engineering are a bit thrust into this role of figuring out, how does that work? I mean, even, like, the procurement processes for some of these technologies are really tough. Legal teams are needing to create new rules about, well, you know, how do we think about software? And there's been 20 or 30 years, you have a SOC 2 compliant to this procurement process. Now all of a sudden there's this thing, where can a vendor use my company's data to train a model? Is that valuable? Do we want the data that's on and inside of VPC or on prem? Is that something that's really important to us? And so I think even there's like, the technology diffusion curve, like, what does this technology do? And that's slowing adoption. And there's the who's responsible for securing it? And then there's the how do we get this through procurement in a way where nobody's fired? And so those three things, those three dynamics I think have people are still. Companies, organizations are still figuring it out.

Okay, that's really interesting. And you know, how do you see, you know, you mentioned here, these challenges, Tom, either maybe from your perspective, do you see technical challenges as well that are hindering adoption today in the enterprise?

Yeah, I would.

Say there are three.

Main challenges, and not surprisingly, these are three big focus areas for us as well. One is the adoption of the technology itself. You have a lot of engineers and systems and application builders that haven't had the experience of data scientists or data engineering or ML engineers, and they are now entrusted with building those things. So they need to catch up and learn very quickly. And we're trying to do a very good job at both educating and giving really good tooling and making everything work out of the gate. The second thing is people need to actually make the applications more knowledgeable and perform better. So it's not just, you know, if you could build something by just, you know, hooking up to an LLM, great, you're done. But most applications don't work that way. They need to do support, they need to know about your documents, they need to know about what a process is. They need to connect to your own data somehow. And so that means you have to somehow provide them with knowledge. And that is very difficult today. Vector databases and Python specifically make it a lot easier, but it's not enough. We have launched a system today, today, I think, or this week, that allows people just load document and run basically text queries and, and so on, and answer questions and complete tasks with their own data. But that's still a challenge for a lot of people who don't use Pinecone. And the last thing really, you know, underscores what Tavar said. It's not enough. You can build an application that will blow the CEOs like socks off, but you would still need to be SOX compliant and HIPAA and, and GDPR and, you know, have jump through all the fiery hoops. And a lot of them are actually complicated questions, you know, and so it's, it's still going to be hard. So, you know, this enterprise readiness is, I think, a big area of friction for buyers and investment for companies who work in this space.

Okay, that's great. And maybe I think it's time as well that we can discuss outside of challenges in deployment. Let's maybe switch gears and discuss the technical trends that we see within Generative AI today. We've seen LLMs become multimodal and LLMs providers improve their different product experiences. Think GPT 4.0 with the voice mode and the data analysis mode. Claude 3.5 with artifacts. These are great, you know, UX UI innovations as well. However, we haven't seen necessarily like a leap in intelligence comparable to the transition from GPT2 to GPT3, GPT3 to GPT 3.5, and GPT 3.5 to GPT 4. Right. Do you think that we'll see this other massive leap in intelligence soon? If so, what do you think will drive this leap? If not, why not? Nick, I'll start with you here?

Well, you know, actually, you know, I was gonna, I was gonna yield or cede my time to answer to, to maybe, you know, I, I'm not as, I'm not, I don't have the most expertise on sort of the LLM architecture details and, and so he might have a better answer. But you know, before he jumps into that, I'll just say my prediction would be that regardless of how some of those next wave of leaps play out, I think we're gonna see a lot.

More.

Juice being squeezed from different ways of combining multiple models together. So more agentic workflows. I know that's, that's very, a lot of people are talking about that right now and I think that's also going to enable kind of harness, better harnessing or extracting more value from smaller, more specialized models and I think we'll get more, more system AI systems that are using a lot of more specialized or small models potential likely in sort of agentic patterns to improve overall performance of these systems. But anyway, to answer your question directly, I don't feel like I have the best expertise.

So let's switch gears then. Ido, I'll let you take that on.

I'll offer a contrarian view that I actually don't. Models are going to, the current way we think about models are not going to improve dramatically in will qualitatively change what can be done with them. Okay. For multiple reasons, the like physics kicks in at some point on how these things are trained. There is the economics of doing that. You know, when model training a top notch model was quote unquote only $10 million, which was way out for any tiny company. But a lot of medium and large companies have that capacity. When you need to invest a billion dollars to train a model, suddenly.

This.

Is really kept to a very, very small set of players. And so, and for reasons that you know, I, you know it's going to be take a long time too go through. I, I personally think we're reaching a level of saturation that it's going to be hard to qualitatively change. And yes, there are improvements but to me product wise if some accuracy measures goes from 80 to 85%, it might be a tremendous engineering feat and, and it could be a big difference. But it's not a qualitative change. You didn't go from you know, 15 to 80 and you didn't go from 80 to 100.

Yeah.

And yes it's a better model, but is it qualitatively different? I don't think not in terms of the applications and use cases. What is going to change is that we are going to make these models a lot more dynamic, knowledgeable, plugged into the right places and find ways to make them basically improve the ways that they train and make them 10 to 100 times bigger. And that's happening, but it's not the models, you know, and the, the, the, the way you think about a language model, it's going to be completely different. Like so a quality, a qualitative change has to happen for that curve to break, basically, in my opinion.

Okay, so models are going to become more useful, but not necessarily raw intelligence wise. They're going to have a simple, a jump in.

No, no, they, they will, but. Well, it's, it's hard to explain. I might, I might have to stop here. But I will say that I don't think the, the models, the way they are trained today are I think hitting an, an asymptote. An asymptote that we're not going to improve much on.

Okay, and Tom, I'll let you also react to here. How, how do you view the next wave of LLMs?

Yeah, I think there's a bifurcation. You'll have large language models that will cost like a billion or 10 billion, and you'll have a handful of companies that are training them and I suspect they'll probably be government subsidies for training some of these massive, massive models because they're of national importance. And then you'll have smaller models that are purpose built, that are much more accurate, that are running on your mobile phone, that are much more. And so you have this basically bifurcation and it'll be used for different use cases. You know, query will come in, there'll be a classifier that says, have I seen this kind of query before? And if I have, it probably goes to a small language model to optimize for a particular task and if it hasn't goes to a large language model, it's basically a big consumer search index, effectively. And so I think that's where we're going. We call these things constellations of models. We're starting to see enterprises actually develop some pretty sophisticated models. Layers on top, there's like caching layers in order to reduce costs. Because if, let's say you were working on a translation software company, saying the word for hot dog in Russian doesn't change. And so you don't need to go back to the index every time, you can just cache it. And so that's the way that I see it. I think multimodal is kind of the. I think the two big waves of the day is multimodal, meaning use text to produce video or vice versa, or learn from video and use that to improve text models. And then agentic systems which Nick and Ido have talked about already, which is basically chaining these things together to act on behalf of humans. The multimodal technologies are new. I think you can see it with Clauds artifacts starting to work, people creating three dimensional models of particle physics with just a handful of sentences and then it's rendering. It's pretty cool. And then on the agent stuff, I think the compounding error rates through multi step processes is really hard to manage today. And there are lots of different architectures there, there are suggestor critiques, there's adversarial networks that people are starting to use. There's the use of the suggestion with a classic classifier in order to minimize the error. And that's all research today. It's very, very early.

Yeah. And one thing I think that is kind of we discussed this behind the scenes, right? Which is, you know, are we reaching peak saturation to training data for example, as well, like when it comes to kind of training the next generation of models. So, and you hinted at this either like saturation of training and like saturation of models today. Maybe walk us through the, your, your kind of thinking here a bit as well, in more depth. I'd love to learn more kind of how you see the next wave of training will happen. What do what, what needs to be true to get incremental improvements on models?

Well, I can't talk about it because we're working on it, but.

Tom, maybe I'll let you hint in here as well.

Yeah, so I missed a question. What was it?

Yeah. So do you think kind of we're going to reach saturation on training data when it comes for models today? And so what is the way out for training models?

Yeah, I think. Well, okay, so llama3.8 billion parameter was really interesting. So let me take that back. The way to think about training, one way of thinking about training is there's a cost optimization function here, which is how much data do I need and how many compute hours do I need to train a model. And there was a paper produced called Chinchilla which created a heuristic and algorithm to determine exactly what that was. And Meta decided on the llama3.8 billion parameter to go way beyond it. Where they spent, they trained it on 15 trillion tokens. And the performance there was actually it was very expensive to do, but the performance was really good, especially compared to larger models. And so I think we're sort of figuring out how do we squeeze as much juice from the lemons that we have and exploring that surface area. We're probably in a local maximum. The other dynamic, and Ido and I were talking about it before that Llama model was trained on about 15 trillion tokens. Our estimate is just somewhere like roughly 20 trillion tokens on the Internet. And the question is, okay, well, we want to train a model on like 100 trillion tokens or 500 trillion tokens. Where does that come from? And I think that's. That's tbd. Nobody knows. Right. It could be video, it could be content that's yet to be created. It could be synthetic data where computers are actually identifying where models are weak and then producing data to supplement it. So I don't have a great answer for you.

Okay, so I think we'll find out and then maybe if we kind of look at, you know, Nick, you mentioned this earlier today, earlier in the conversation that, you know, we're still in experimentation phase for a lot of tools and, you know, but all three of you hinted at kind of the deployment challenges with generative AI solutions today. You know, a lot of the infrastructure technology to enable fast deployment of LLMs, think monitoring, debugging, retraining systems, all of that still needs to be built. So how do you feel maybe the maturity of the LLM middleware ecosystem today or like the generative AI middleware ecosystem today? So, Nick, maybe I'll start with you here as well.

Yeah, I think it's similarly early. I mean, and the other thing I would say is I don't. I think a mistake people are making is thinking it's conceptually a completely different thing from like, let's say, the traditional MLOPS challenges we've all we've had over the last five, 10 years, I think. You know, I hear some people think, well, LLMs and Gen AI make it so we don't, we don't have MLOps problems anymore. It's a different set of problems. And rather I think it's more like the, the Gen AI use cases take all the same problems we had with traditional mlops, but they just, they just exacerbate them. Or now, now it's like if we're playing a video game, we've activated the hard mode. So, you know, you have, you have a lot more intensive compute resource requirements, infrastructure requirements. It's GPUs, not just CPUs. You need to scale these things. That comes with a whole bunch more cost you need to deal with. I mean we have customers that have gone from deploying again for prototype use cases, traditional predictive ML to now gen AI and all of a sudden their compute bills are going out of control and now they're, they before they didn't need to worry about things like elastic auto scaling for inference, now they do because the GPUs cost a lot more. You know, you talk about, talk about model governance, model monitoring, that's all based on the fundamental characteristic of any sort of predictive AI system, that they're probabilistic. And so their behavior can, they can have unexpected failure modes or behaviors can change, you know, or their performance can change or the world around them changes. Well, Gen has a much wider surface area of unexpected failure modes than say a traditional predictive ML model. And so now the set of things you need to do to check and guardrail and the set of human, human processes that you need to put these things through as you move them through a sort of a quality assurance process, that's all much more complicated. So yeah, I mean look the sum it all up, it's like all the same problems we had with MLOPs are now incremented or exponentially harder for Genai. And I would just add, I think a lot of these challenges are not. You asked about middleware, I'd say a lot of these challenges are not strictly technical. They are business process challenges. They are people in process orchestration challenges. If we're going to put a gen AI model or system into production in an enterprise that might touch customers, especially in a regulated environment, a bank, an insurance company, a life sciences company, What checks have been executed to ensure that is compliant, that is safe. These are. Everyone that we work with is still trying to figure that out.

Okay, great. And then Thomas, from your perspective, you're investing in this space, right? Like you have a very elevated, you know, bird's eye view over the space. I'd love to kind of see how you see the kind of infrastructure space for LLMs evolving today.

Well, there's, I mean, lots of exciting things happening, right. Euro's building a great business. Vector databases are really important. We're investors in a company called Superlink, which is called a vector computer, which allows you to take structured and unstructured data and put it together in a unique way. And so I think where, you know, but it's really early in this ecosystem where as we talked about, buyers are still trying to figure out exactly what it is that they want to build. And until they know what that is, it's hard to have a view on exactly what the ideal architecture is. Then you have, you know, OpenAI and the other big companies also deciding strategically what areas of infrastructure do they want to play with.

Right.

So OpenAI released an open source evaluations framework to help software engineers understand exactly how well a model is performing before they release. A little bit like a testing harness. And that was a move for them into a category where there are a lot of startups already. So it's, you know, I guess for startups, I guess the way I'd put it is you're dancing with elephants. They're these big companies who really care about these markets. Microsoft has a $5 billion run rate business in 18 months. OpenAI is north of 3 billion. And you have others that are growing pretty fast and they're still deciding what parts of the stack they want to play with. You look at OpenAI's acquisition of Rockset last week, I think, and open questions about what did they do with that technology and that team. So the chessboard is not set. There's still a lot of moves to be played and you have to navigate. I think startups have to do what they do best, which is be nimble and navigate the ecosystem as it's rapidly forming.

Yeah, brilliant. And Ido, you know, you're building a company that's at the center of the infrastructure space and LLMs, you know, walk us through how you see this space evolving over the next year. And I don't want to, I don't want to hear like product roadmap, that's not the question. But I would love to see kind of how you think the ecosystem will evolve in the year to come.

I agree with Tamash100. I mean this is the, it's a very dynamic space and I think that the, there's going to be one truism that's going to still prevail here, which is. Companies and enterprises specifically are going to find AI more production ready and useful very soon. We are working on it, others are working on it. The elephants in the room are working on it. It's obvious that this needs to happen. And so exactly who does what part of the stack and who gets to own what piece of the piece I think is anybody's guess. We of course want to have the biggest piece that we can own out of that. It's obvious, it's like any other. But I'll be intellectually honest and say that nobody can really Tell.

Yeah, and I think this segues to my next question really well, because everyone's working on this problem of deployment, making sure that models are very productive. You know, I think Nick mentioned earlier that we're at the peak of inflated expectations, but we will reach the slope of enlightenment at one point in time here. Right? And once we reach critical massive adoption, if generative AI, you know, reaches critical mass in terms of adoption, how do you think it will shape the nature of work or, you know, the. What does that future look like? So maybe. Thomas, I'll start with you.

Sorry, just going off mute. I think, you know, the way that we think about it is there's lots of toil within work. There are lots of repetitive actions that we do. And each, each discipline has its own right. Within sales, there's the role in sales, development of understanding leads and qualifying leads and deciding which ones to pursue. Within the world of legal, there's paralegal work, which is the administration of documents within the world of accounting, data entry across different kinds of tax forms that have different formats. And within the world of software engineering, it's remembering which arguments go in which order in a function call. And I think the future of work with AI is having computers solve a lot of those rote tasks and work alongside of these copilot systems that we have. And today, the productivity gains, at least from some of the early companies, suggest it's about a 50 to a 75% productivity gain. Maybe it's a little bit aggressive. Computers will not love each other. That's true, Luis. But I think if we look to the future where work is actually automated and we can delegate tasks to computers the way that we delegate tasks to humans. The best analogy is looking at automobile manufacturing lines where robots have replaced a lot of the labor. And there you see about a 275% improvement in productivity, where one robot can produce about the output of 2.7 humans. And I think we should be able to get close. That's not based on anything except a rough guess, but I think order of magnitude we should be able to get to a place where a lot of this toil, this rote work that is unappealing is automated. 100 years ago in America, there were 4 million human dishwashers, people who wash dishes for restaurants.

Wow. Okay.

And then the. The first robot introduced in most people's houses was a dishwasher. And now we don't. You know, the chores that we give to our children is loading the dishwasher instead of washing the dishes. I think that's that's a very good analogy for future work in white, color work.

It's very, very fascinating. And Ido, I'll let you here also react. How do you see, you know, once Pinecone takes the biggest part of the pie as much as possible, how do you see the view, the future of work here?

Look, I mean, I think we're producing great tools. There are, I think we're very far from any model or set of models or agents completely replacing somebody's job. I think that's, that's, that's very unlikely unless your job is so menial and, and just whatever, like, but except for extreme cases, okay, all the lawyers and doctors and accountants and analysts and this and that, they're not going away. What they are getting is fantastic new tools that make them work more efficiently, more correctly, you know, offload the grunt work that maybe they, they don't want to do. And that's going to happen. I mean, I'm 100% sure of that. And history shows that when you have something like this, it actually improves overall productivity for society, which ends up being good for everyone.

Okay, great. And then with productivity, when the paradigm changes for productivity generating technologies. You mentioned this, Thomas. The chores one used to give to their kids was washing the dishes. Now it's loading the dishwasher. That skill has changed, right? The skill from washing washing dishes to loading dishwasher. What do you think the skills needed? Like, what would your advice be for individuals today to build the skills needed to kind of adapt to an era where generative AI is widely adopted? So, Nick, I'll start with you on that question.

Yeah, well, I think, I mean, building on what Tomash and Ida both said, If Genai plays out the way everyone's expectations hope it will, it's going to be leveraged for people. It's not going to replace people. And so in any given industry or any given job someone has today, there are a set of things that can be delegated away, but there's going to be a core essence of that job that is the insight part or the creative part or the architecture part that is going to be, that is going to remain undelegatable to one of these AI systems. And so I think, yes, if you're a software engineer, then you don't need to remember the order of the arguments to a function call, but you do need to be able to understand a customer requirement. And based on that understanding, design a software architecture that appropriately reflects the abstractions. You need to model the customer's problem. Domain and you know that. So the conceptual or more insight based parts of any job are going to be increasingly important. You know, if you're a graphic artist, you'll still need to be able to have a picture of something in your head that you want. And then you can interact with, you can coach, you can instruct an image generation system to create that, but it'll be getting a computer to create what you are picturing in your mind's eye. So maybe the technique of how do I use Photoshop to compose this thing I'm picturing, maybe that will go away. And just combining that thought with your question a minute ago about what this all means for the future work.

The.

Interesting potential dynamic I am thinking about is what will this mean for a whole wave of disruption that flows across every services industry. So if you're a consulting firm, if you're a graphic design firm, if you're, if your business is fundamentally a services business, I think that you are at risk over the next, you know, five years of being disrupted by a much more efficient, much more scalable, much more high leverage model of executing that business.

Okay, that's great. And Tom, I'll let you also react. What do you think the future skills to adapt are?

Well, I remember when all the search engines came out, right? Google, Metaweb, Yahoo, AltaVista, and you needed to figure out exactly how to use them well and the people who could use them well. We're a lot more productive at work and so that's where we are. So I think prompt engineering is probably the skill of the day today because if you can do it really well, you can produce a blog post, you can produce marketing copy, you can produce images, you can produce code. And those marginal benefits, if you're 50% more productive than a coworker or competitor, you will be promoted. And that's sort of the name of the game. I think all software is sold because it's a promotion in a different form. Somebody buys the software and champions it because they can be promoted. So I think prompt engineering is the thing that, or the skill that is the most broadly applicable. There was this awesome study of, I think BCG or McKinsey, one of the big consulting companies, studied the way that, yeah, it was called Centaurs and Cyborgs. And one of the most interesting use cases there was asking the AI to simulate a potential software buyer and do the be the foil in a conversation. We're seeing this in education too, where students are creating their own personalities to talk about how to solve a Calculus problem or chemistry problem set. So I think that's, that's, yeah, that's probably the broadest and most applicable use case today.

Okay, great. And then Ido, I'll let you finish us off with, you know, how do you view the state of skills in an era where adoption has matured?

It's very hard to, it's very hard to, to know to be honest. And I might not have the, the best like a very strongly held opinion on exactly what's that's going to look like. I remember also when search engines came out like people imagined all sorts of professions that ended up not existing and, and so on. So I think prophecy was, is on, on this topic is, is a bit hard. I think it's, it's obvious though that this is a, like a set of very powerful tools are going to be produced and professionals who don't master those set of tools are going to be left behind. You know, if you're, you know, if you're that lawyer that rejected the use of a PC and kept using your, you know, you know, whatnot. Yeah, you didn't stick around for a long time.

I couldn't agree more. And I think we have time for one question to take from the audience and I'm going to take this one because I think everyone here hinted at the potential improvement of agentic systems in the future. So maybe I'll ask it here, just looking for it. Do you see? Here it is. Do you think that we might see agentic approaches increasing in effective ineffectiveness to the point that it reaches the level of predictive? So for example, can you reach a point where agentic systems are so good that you can predict with accurately that they will work well most of the time? Are we headed to that type of future? Because you know, Thomas, you mentioned the compounding error rate across the different steps of the agentic workflow. I'd love to see where you think agentic systems are headed in the near future. So let's start with you maybe Tom.

Yeah, I think we. Okay, so I think let's put it a different way. I think we'll get to a place where we will brute force innovation. In other words, like we'll kick off massively parallel jobs that are trying to figure out what is the next generation chemical compound. And it's not that the systems will predict, it's that we will just throw so many computers at it that at some point in time they will solve, they will find a solution. And so, and we'll just, you know, it'll discover huge solution space and one of the answers will be there. I, I don't believe computers can predict very effectively. And you look at classical machine learning, they've had a, I mean they've had a really hard time. They can predict things that are very stable and repetitive and cyclical. And Google made a really beautiful business model based on very predictable patterns. But for things like core innovation or creation, that is not just recombination. It is just not the right technology to solve that.

Okay, Nick, I see you're nodding in agreement. I'd love to see your thoughts here as well.

Oh yeah, I was just going to agree with Tomash about the, you know, brute forcing innovation. There's a hedge fund we work with that's doing that exact thing using Gen AI to come up with candidate investment algorithms. And then of course it's easy to, or something you can test and check each one. And their point of view is, look, if we generate 10,000 and 900 or you know, 9,999 are bad ideas, but we get one good one that's, that's worth it. But I think, I think the, that brute force innovation, you only get the ROI for these areas where the innovation has a big payoff and you can kind of justify the investment to search a big space because the vast majority of the ideas will be bad either.

I'll let you finish us off here. How do you feel? How do you see agentic systems evolving?

Well, I'm both in support and violent disagreement. I mean, I think brute forcing doesn't work. Exponents tend to be exponential. And anybody who wrote like a five nested for loop knows that it gets pretty terrible pretty quickly. And so yeah, I don't think this is solved by just throwing more hardware at it. For the love of God. We've been throwing literally billions dollars, billions of dollars worth of energy on this thing. And you know, we didn't solve everything, so I don't think that's the way to go. I, I do think that some innovation comes from those systems that then agendic systems can actually discover interesting things because humans also can produce exponential amount of compute in, in our own heads. And the people who do research are also energy bound. And so whether we do this in brute force, I don't think so whether it actually produces interesting insights and innovation, I think 100%.

Okay, I think this is a great place to end our chat. I want to make sure everyone send as much love as possible to our speakers today. Thank you so much, Thomas. Thank you so much. Ido, Nick especially. Ido and Nick are both on vacation in Greece and Italy, respectively. So I really appreciate them making time out of their, you know, precious vacation times as, as leaders of their organization. So I really, really appreciate your times. And thank you so much for everyone who attended. And again, huge round of applause for our speakers and see you on the last session on our closing session. Thank you all so much.

Thank you.