The Where, When, and How of AI with Theory Ventures, Open AI, MotherDuck and Lamini

Hi everyone, my name is Tamash Tunguz. I'm founder and general partner at Theory Ventures. Thrilled to be here with you all and our illustrious panel of guests. Today we're talking about how data and AI are changing the way that applications are built. We've got an incredible group of people here, starting with Jordan Tigani and Sharon Zhao and Maggie, and would love to just kick things off by having them introduce themselves and we'll jump right into the agenda. Jordan, do you want to take it away?

Yeah, sure. I'm co-founder and CEO at Motherduck. We're building an analytics engine based on open-source DuckDB. I helped create Google BigQuery, led that for almost 10— or was part of it for almost 10 years before starting Motherduck about a year ago.

Hi, everyone.

I'm Sharon. I'm the co-founder and CEO of Llama and I. We're making it easy for enterprise customers to be able to fine-tune their own LLMs. Most recently, I was computer science faculty at Stanford teaching generative AI, where I also got my PhD with Andrew Ng. I teach about a quarter million professionals online generative models, fine-tuning LLMs most recently. I'm very excited to be here today.

Hi everyone, my name is Maggie Hott. I'm on the leadership team at OpenAI, in particular overseeing the team that is bringing ChatGPT Enterprise to market. We launched about 10 days ago. Prior to this, I spent 2 years at Webflow building out the sales team there. And then prior to that, 6 and a half years at Slack from the very first founding sales hire out of headquarters all the way through acquisition to Salesforce.

Incredible backgrounds. Well, I'm really excited to talk to you all today about the way that data and AI are changing the way that applications are built. It's just been incredible last 6 months. Look at the rapid pace of innovation within LLMs, how they're really transforming a lot of the different ways that people are using software. And maybe Jordan, we can start with you. At the data layer, you built— you're commercializing DuckDB, one of the most popular databases now in the world, super high performance in-memory database. How are you seeing AI change the way that applications are built and the way that people are using data?

So to start out with, I was a little bit of like a resistor thinking, well, this isn't really going to change our world. Maybe it's changing a lot of other things. But then I was talking to somebody who was giving a course on data thinking at Princeton, and he was teaching people how to ingest data, how to query their data, and then how to visualize their data. But he wasn't teaching them any programming languages. He wasn't teaching them SQL, and he wasn't using higher-level tools that can automatically kind of do this stuff for you. He was just using ChatGPT to generate Python code to ingest the data, to generate SQL statements, to generate the than the Python visualization stuff. And I'm like, wow, like, you know, and this was like, this was a few months ago when ChatGPT had just been out. And I just, you realize that like, yeah, this is gonna change who can do this stuff and also change the skills that people are gonna need and the tools that you're gonna need to build. So I think that's one side of things. And the other side of things is just the, you know, developers, people who are building things with, that need access to data are going to be changing how they access that data. And I think the V1 is like the vector databases, but I don't even think we know necessarily what V2 is going to look like because, OK, you get those vectors out. OK, what do you do with them? How do you actually access your data stores and relate that to what you're getting out of the models? So I think there's a whole big area of unknowns and exciting possibilities there.

Yeah, I mean, one of the things you touched on there that I think is really important is how much LLMs are democratizing technology. It's really much easier for someone who's maybe not be as sophisticated with SQL to suddenly start asking and answering questions of their data in ways that would have taken them 3, 6, 12 months to learn beforehand. It's an incredible— phenomenon. And it's being adopted broadly within the enterprise, right, Sharon? I mean, you're seeing this. We're seeing this just kind of across the board. You look at the total number of customers that are starting to use Microsoft's OpenAI technology has 3x, I think, in the last quarter in the reporting. How are you seeing enterprises start to adopt some of these technologies and how are they managing it? There's a lot of technology here. It can be complex. There's tuning. There's lots of different components to it. What are you seeing in the market at Laminate?

Yeah, I think even just zooming back out at a technical level, I think actually public data is running out for all, all of what LLMs can take advantage of. So actually the next frontier for LLMs is in enterprises. And I believe the best LLMs for this next, next wave essentially will be enterprise LLMs. And the enterprises that I'm seeing that are, you know, at the forefront of this, they have started to adopt techniques around fine-tuning their models to personalize models for every single user and every single customer of theirs where the data doesn't actually leak between those customers, but the model is highly personalized according to their access controls for data and just any kind of complex— any kind of complexity around their data slices and use cases. So that's what I'm seeing, and it's honestly really exciting to see this because this will to further democratize all the possible use cases that can be out there, far beyond what I can, I know, even imagine.

Yeah.

So, OK, so what we're starting to see— we're meeting lots— I'm a venture capitalist. We meet lots and lots of different businesses. They're starting to offer these software that's enabled with AI. And there's a question that we're sort of wrestling with in the ecosystem, which is who— is it the vendor who's responsible for actually optimizing Or is it, are there people inside of the business that are subject matter experts? Are you seeing any trends on which side is going to be responsible or mostly responsible for optimization?

Yeah, so there's definitely this tension between the two where AI expertise, it might take a couple hundred top AI researchers to be able to train some of these models such as at OpenAI. However, I believe that, and this is based on my experience training these models, it's actually the domain experts will be driving the best models out there. It won't be people like me who can actually do all the model training, et cetera. All that infrastructure will be built out such that more people can actually define the trajectory of these models. And as we saw with ChatGPT, with more people being able to use these models, they thought of way more creative, creative different use cases than like I in a silo with a few other AI researchers can think of. So I think the trend we're seeing is really, you know, the, the typical developer, the infrastructure developer, even the front-end developer, all these developers are able to— who are able to access that data and be able to manipulate that data are the ones who can actually help develop the trajectory of these models and do the fine-tuning and actually build out this next generation of LLMs for the enterprise.

Yeah.

That's a great segue. Maggie, you're in the center of the world at OpenAI, basically catalyzed a big chunk of this movement. You recently launched OpenAI for Enterprise. Can you tell us what was that like? What, what are you seeing in the market? What's— what sort of revelations can you share?

Yeah, absolutely. So I will actually break this down into 3 things. I'll share with you all a bit about our journey to building ChatGPT Enterprise over the last, you know, few months. Um, I'll also share with you some of the learnings and kind of use cases and, and what what we learned from working with our design partners. And then finally, I'll share a bit about what is coming, which we're really excited about. So I was hired on to OpenAI in April, really with the mission of building out the go-to-market team for ChatGPT Enterprise. We knew it was something that we were going to build. And that was because when ChatGPT launched early last December, all of a sudden we started seeing millions and millions of knowledge workers using this tool. Every single day for their jobs. But then you have kind of the age-old security, privacy, IT side of the house that is saying, whoa, we don't want our corporate data being put into these models and trained on these models. So we put our heads together and said, let's build an enterprise version of this that has things like data training turned off, SSO, SCIM, I mean all the enterprisey features, SOC 2, CCPA, GDPR, all the acronyms. So when we set out to really launch this, what we decided to do was to select a pretty big variety of design partners. So for example, Amgen, PwC, Bain, Samsara, we decided to go as wide as we could with different industries and use cases, quite frankly, because we wanted to see how healthcare uses this tool, how does banking use this tool, how does consulting use this tool? So we worked really closely with them behind the scenes, of course, you know, under some pretty mega NDA lockdowns, for the last few months to understand what is it that the market wants to see for a V1 launch. Throughout this, there was a really big learning that came out to us, and it was the importance of developing alongside your customers the idea behind really strong use cases enablement training. I think when we started off, we just kind of assumed, oh, everybody knows how to use ChatGPT, but then we realized everybody uses it in a different way. That they've kind of created themselves. So we've been thinking very deeply about how do we work with these design partners and now the world to develop very specific tailored use cases to their industries, to their personas. ChatGPT can be used by every single vertical, every single company, every single department within a company. It's just mind-blowing. So that was a really big learning for us, was really invest in customer success type work early on. We set up Slack Connect channels. We asked these design partners to have weekly syncs. We sent out surveys to all of their beta users. And so we feel really good about what we launched last week. And we've also been quite public with our roadmap and what we're building here, which I think is also quite unique. Normally companies are a bit cagey and we've shared pretty broadly that the single biggest thing that we are working on right now is this idea of customization. So imagine being able to have your entire company's corpus of data or all of your own personal data infused in with ChatGPT Enterprise and basically being able to query, you know, all of my top sales deals, what were common trends across them, and having that all come back for you. The other thing that we are also building right now, which is really exciting, is a business version. So more of a self-serve version for SMBs. We realized, you know, we went out of the gates with enterprise, but a lot of times SMBs just don't quite need all the enterprise features that we launched. So there's just been pretty huge demand from SMBs, from startups, from founders to be able to adopt this. The use cases are endless. Code Interpreter, which is now renamed Advanced Data Analytics, is probably the most powerful tool I've ever seen in my life. You can essentially ingest in hundreds of thousands of rows of data or spreadsheets. I think about it as a super calculator. I use it personally for looking at leads, looking at, you know, sales data analysis, opportunities, win-loss rates. Everything, and it basically just synthesizes anything that would typically take someone a day to parse through in minutes. And then down to the bare bones of like recapping customer calls, feeding in transcripts from Gong and having it analyze that and then write a next steps follow-up email that is like beautifully perfect, personalized and tailored, which would have taken me half hour per email before I can now do in 10 seconds.

That's amazing. I think the point that you hit there, that solution sale where you have to take something that's a general purpose real innovation and really solve an end customer problem, that's what we're seeing in the ecosystem today. It's like people were really excited about the technologies and industry, but we need to figure out, okay, what is the problem that the end user really needs to solve? Sharon, Jordan, are you starting to see, like, are you focused on individual personas when you think about taking your products to market or seeing different use cases arise?

I think, you know, there's the sort of developer persona, and I think those I think the way those— I mean, as you mentioned, that everybody uses ChatGPT in a different way. So the developer is going to use it in a different way than an analyst is going to use it, and which is different than a data scientist. And I think, I think, yeah, there's going to have to be different, different tooling for, for each one of those.

We're hyper-focused on the developer. So the software engineer, someone who can actually deploy a real app in production. And we're just really focused on getting these basically production use cases out. And we believe that the software engineer who sits most closely to their production apps will be the one to do that. And we've been finding that they're able to leverage data really effectively and efficiently. They're able to hook up their internal systems, whether that, whether that be recommendation systems or search systems into these models as well. So they're, they're, they're very capable at pulling together all of these different APIs and continually improving these models, um, on a schedule. And so we're hyper-focused on this persona. Um, but I'm also really excited to see just so many people be able to use this technology.

Yeah. Yeah. That's been a big change. I mean, 10 years ago, the way that I thought about the data stack was you had systems that produced data and then they would put it, put it into like cloud data warehouses or data lakes. And then there were 3 consumers. There was the BI use case. Exploratory data use case, and then there was machine learning. You're trying to build data products, and now it really seems like those machine learning engineers or that functionality is becoming part of the core products, becoming part of the core workflow. Like Maggie talked about, Gong, analyzing those sales calls, that's part of the product. The machine learning engineers are now starting to be put into the path of production. There's this blend, there's this mix between, okay, you've got a classic software engineer who's used to shipping code with CI/CD and the classic workflow, and then you have machine learning experts and bringing in different technologies, whether it's data caching or customized models or off-the-shelf APIs, and you're putting it into production, it seems like there's a merger there of two personas that historically have been separate. The landscape is sort of rising up to, how do we enable those people to be as effective as they can be using things that maybe their expertise is sort of out of their domain? So you need a higher level of abstraction. Is that what you're seeing in market, Maggie?

Yeah, absolutely.

Yeah.

And I think that's incredibly important, actually, merging those two together. Because historically, maybe to give a fun anecdote, one of my friends used to lead engineering at Uber. And so 1,000-person engineering team. And what he found was the machine learning team was building models in Python notebooks that were obviously then had to be sent to the production team to then ship it into all of our apps. And one time, because Uber was moving very fast, let's put it that way, they had like just fired the person who ran this other model. So they had to literally go find the person's personal laptop, open up the Python notebook, rerun the data on these like a few states where they had to like have an updated model and then somehow rip that model out. And thank God they found that laptop and put it into production and now it's shipped on our phones. And that is an insane process. That's super brittle. And I guess another anecdote is just It's really hard to get some of these systems up. In machine learning, a lot of AI people are like, "Yeah, we push the performance of this model and we define performance as accuracy or accuracy along these general benchmarks." Maybe it's 6th grade science questions or something. But a production software team will look at that and say, "That's not performance to me. I also care about the speed, the latency of this API call, you can't give me a better model and give me 100x slowdown of this model because I'm putting it in an autocomplete type of interface. That's not going to work. I think merging these teams together, bringing them together is really important to actually shipping these models in production. I think historically it's largely been this R&D team with a really thin line to production and oftentimes failing that. Only very few teams could succeed at that before, but I think now many more teams will be able to succeed at that and it's just really exciting for AI more broadly.

Yeah.

And I think as those teams are merging, there's, you know, there's this sort of interesting debate, which is what is the lingua franca of, you know, what language are people going to be writing in? Is it, you know, like, is it Python or is it English? And so as people start to be able to use just English natural language to do some of the tasks that were in the other camp, I think that also can help accelerate that merger, even if sort of maybe fully getting towards, you know, you describe your program as sort of like an English language paragraph and have that turn into a production system. Maybe that's a little bit further off. But I think as you start to kind of peel off workloads, it can just help those teams merge.

That's the thing that's so exciting. I mean, think about Microsoft Office, 330 million paying subs, whatever. Let's say a third of them use Excel. All those people want to be able to access some more sophisticated data. Within the organization, they want to analyze it. Like you said, historically they've really needed to learn either VBA or SQL. It's a language not everybody really knows, but now if they can start to speak English, you really have the power of data now being exposed to, I don't know, 10x, 100x greater the population. It's pretty incredible. Just as we've seen, I mean, what we're talking about on the technical side of an organization, you really see a big transformation in what's happening. Maggie, you're a sales leader. How is— how are you seeing LLMs and these applications change the way that companies are starting to go to market? What's going on in the front of the house?

Yeah, I think it's best to start with a bit about how sales has changed over the last few years, because every year it seems to be there's a new step. And I think the short of it is that AI is going to fundamentally change how the sales role operates and looks. And it already has. We're already well into that. 2017, 2018, it was pretty easy to prospect into companies. You know, prospecting is kind of the bread and butter of any sales organization to generate opportunities and customers. Then all of a sudden you have these tools of Salesloft and Outreach, which come out and just take prospecting mainstream. So then all of a sudden sales reps are sitting there saying, what do I need to do next to get in front of buyers? The next step was personalization. So years ago, personalization wasn't even a thing. Last few years it's been all about how do I research this company, this customer, this person? Did we go to the same school? What can we kind of connect on together? Well, now LLMs have made it so research and personalization can be done in less than a minute. In less than a minute, you can have ChatGPT or Jasper or any of these other tools that are out there right now go write an email that sounds very personalized and tailored but was actually written by an LLM. So now this kind of next era of sales and sales orgs has to really be thinking, deeply about how to interact with the customer. No longer can you get on a call and ask somebody, you know, a million discovery questions. You have to show up to that call with a strong point of view with how your, your product is going to save this company money or solve their pains or whatever their mission is. And the beauty about LLMs is now they can help you do that research in advance. So back in the day when I was a sales rep, I used to spend 2 to 3 hours before a big meeting, even more researching and prepping and practicing my discovery questions and coming up with a value hypothesis. Now that time can be significantly reduced to maybe 20 minutes max, and you can literally have the LLM be role-playing with you of the questions that you are going to ask and the value statements that you're going to bring. And now I have, you know, LLMs also on the other side of it are going to save so much time for the sales professional in terms of administrative tasks. No longer are the days of writing, as I mentioned, long follow-up emails or logging your notes into Salesforce. You can literally feed, as we talked about, transcriptions from Gong directly into there and use prompt engineering to write these beautiful summaries. There's all the new tools that are building slide decks for you, which is wild. So you're able to save on the administrative side of things, but the sales rep in this day and age, and sales teams have to really step up with their value selling.

Do you think that changes the structure of a sales org at all, or do you think we stay with like the SDR, AE model?

I think we stay with it for now.

Yeah.

And then I think AI, I think sales teams and companies that adopt AI are going to significantly outpace those ones that don't. So I actually think now, like, the common sales rep should be doing everything they can to actually learn prompt engineering to learn how to leverage AI in their day-to-day to make them so much more efficient. I think maybe someday the model changes, but I don't think we're at that day just yet.

So we're all salespeople, right? If you're head of sales or you're a salesperson, you're a venture capitalist, we're all selling something. I think these tools, like you said, they open up like enrichment, discovery. How are you using LLMs in your day-to-day in order to improve your life? Are there tools that sort of stand out for you where you've seen a complete game changer? Anything kind of come to mind?

Are you asking me?

Yeah.

I think certainly people that are interacting with data, it's changing their lives. One of the things I do a lot of is I do a lot of slide decks and things like Midjourney and generative AI make it so much easier to have the right image. And yeah, maybe sometimes people have too many fingers or there's ducks without wings or kind wonky eyes, but I think as the technology gets a little better and perhaps like the operator gets a little better at like prompt engineering, you know, those kinds of quirks will go away.

Well, with Mother Duck, I mean, you've done an incredible job making that such an iconic part of your brand. I could imagine getting the duck anatomically correct is really, really important. It is important.

I think one thing that, you know, we were analyzing our data when it came to all our marketing blog posts or like tweets, everything going out. And there was a 10x increase in anything with an image, right? And so it was just much more engaging for people to actually, you know, have impressions on. And so we found that just spending some time on Midjourney was worth more than spending time on the blog post. In any single way, like hands down, just like spend a few extra minutes here prompt engineering or really just like generating an image of your choice and clicking through them versus actually writing out the blog post that no one's going to read. So I think like that was a really big, big lift for us when it came to, when it came to marketing specifically.

Yeah, just piling on, I, those images are super helpful. I was unfortunately sued for using an image where I didn't have the rights, and so I learned a very important lesson. And so now all— most of the images on the, on the blog post that I write are Midjourney generated because now I can be confident that I'm not going to get myself into legal hot water again, uh, with rights issues.

Yeah, go more into that story. I'm kidding.

No, no. Well, so I mean, you know, I was like, I was— I— you want a hero image, right? Like, you look through like the click-through rates and the dwell times, and so I was using Google and there's a filter on Google where you can search for things that are Creative Commons licensed. And so I found an image on the Google search results that ostensibly was licensed with Creative Commons. Turns out actually somebody owns the rights to the image, which I was notified later. And so now at least with like a synthetic image, you can be totally confident that effectively— I don't know what the latest legislation is, but nobody owns the image, or I at least will the image. So that's been a huge help. Do you use anything daily, Maggie, that really sort of where you see like a pretty meaningful improvement in your day-to-day?

I would say ChatGPT Enterprise, but that's not a fair one. I have been the last few years, there's just no tool that's better than Gong. Gong as a sales leader, Gong is getting ramped up as a sales rep or a new hire. Gong for product. I mean, as we've been building ChatGPT Enterprise, we would love to have our product leaders on every single call. It's just not realistic. So we can literally take snippets out of Gong and share all the common feature requests. We can drop in hashtags. I mean, I've— I'm on Gong's website on like from my previous company of speaking about them, but there is just no better tool out there than Gong. So I'd say ChatGPT Enterprise, Slack, and Gong are the 3 tools that I use for many hours every day.

Yeah, that's awesome. We do the same thing when we meet founders. We record the conversation so that we don't have to ask them to repeat. And we're starting to save all that information produce LLMs on top that summarize diligence. And we produced our first investment memo summary from an LLM.

Wow.

And so we can just start to see, like, as we gather more and more of this information, we can accelerate some of the things that are wrote within the org. The other thing that's been really useful for us is blog titles. So, you know, I think about, like, LLMs. I mean, there's incredible things where they're taking the sum total of, like, human knowledge that we've put on the internet or the datasets that we've trained And then they can replicate them. And you think about all of the amount of human effort that's gone into optimizing blog titles. That's all on the internet. And it's all encoded within these LLMs. And so if I take a blog post that I've written, and then I ask one of these models to produce 3 or 4 suggested blog post titles, I actually get 3 to 4 times better click-through rates on the titles because I'm aggregating all that knowledge. So that's been a big, big help for us. Awesome. Yeah. Okay, so to bring us home, would love like quick take. Where do you see the— for your particular company, like where will we be in 3 years? How will LLMs have actually changed your business?

So I'm gonna— a little bit of a cop-out on this one, and it was, it was an answer that, um, a founder, um, colleague of mine, uh, gave, which is before ChatGPT, he could sort of look at look 5 years out into his business and be like, okay, I can see the road and I can see where we're going and I'm pretty sure we're going to be successful. And then ChatGPT comes along and this cloud comes over the road and you can't really see more than 5 feet out in front of you because OpenAI may release something that makes something totally irrelevant or make it so that developers or analysts start to use some sort of other other technology. I think the only thing that I think is clear is that people will be doing things differently. Their jobs will be different. I think, for example, the data space has sort of started to aggregate into these different job titles: analytics engineer, like somebody who uses dbt, data engineer, somebody who munges data, analysts, etc. But As all of that stuff can be done with prompts, I think that there's going to be another— I think there's going to be a reshuffling of those roles.

Makes sense.

I think every enterprise will have their own LLMs. And I think what that'll enable is, in every single market, there are domain experts, for example, like sales. They're domain experts. And what an LLM does is it becomes a scalable domain expert. And so for that market, there will be scalable domain experts for every single type of expertise tailored to every single type of customer and user. And so I find that really, really exciting. So I think that'll dramatically transform job roles moving forward. And that'll dramatically change how basically how businesses need to operate and who they need to even bring on and hire. Yeah.

Maggie, in 60 seconds or less, predict the future for us.

My— I can't comment on OpenAI's roadmap. But what I can say and what something that I'm the most most excited about in terms of an industry that is going to get completely upended is education. Just last week, I was with the CIO of ASU, and he has these huge plans to bring AI tutor bots and AI coursework and really democratize the future of education so that anywhere in the world has equal and equitable access to education. And I think that's going to be one of the biggest impacts this world has ever seen, is everything that's going to happen with AI and education. It's going to be just absolutely bonkers.

I could not agree with you more. I mean, I think the thing that we've noticed across a bunch of different companies is if you have somebody who's new to a space, new to domain, and that could be math or calculus or biology, it could also be SQL or data engineering, if you can give them access to an LLM, the quick feedback loops and a super patient instructor where you can say, like, I don't understand it, or show me this. Or the other day, I had written a line of Bash code, and I had forgotten what it was. ChatGPT, like, what was it that I actually wrote in this piece of code? And explains it to you. It's an incredible, incredible tool. Awesome. Thank you, everybody. Big round of applause for our incredible guests. Thank you so much for joining me today. So grateful.

Thank you.

Thank you very much, everyone.