Tomasz Tunguz Theory Ventures on "AI: The Where, The When, The How" with MotherDuck, Lamini, OpenAI

Ladies and gentlemen, please take your seats. Our program is about to begin. As a courtesy to our speakers and fellow attendees, we ask that you please silence your cell phones. Please welcome to the stage jordan tigani, founder and CEO at mother d, sharon zhao, co founder and CEO at laminai, maggie hott, gtm leadership at OpenAI and tomas tungaz, general partner at theory ventures.

Hi everyone, my name is Tomasz Dunguz. I'm founder and general partner at Theory Ventures. Thrilled to be here with you all and our illustrious panel of guests. Today we're talking about how data and AI are changing the way that applications are built. We've got an incredible group of people here, starting with Jordan Tigani and Sharon Zhao and Maggie and would love to just kick things off by having them introduce themselves and we'll jump right into the agenda. Jordan, do you want to take it away?

Yeah, sure. I'm co founder and CEO at Mother Duck. We're building an analytics engine based on open source DuckDB. I helped create Google BigQuery. Led that for almost 10 or as part of it for almost 10 years before starting starting MotherDuck about a year ago.

Hi everyone, I'm Sharon. I'm the co founder and CEO of Laminai. We're making it easy for enterprise customers to be able to fine tune their own LLMs. Most recently I was computer science faculty at Stanford teaching generative AI where I also got my PhD with Andrew Ng. I teach about a quarter million professionals online, generative models, fine tuning LLMs. Most recently. I'm very excited to be here today.

Hi everyone, my name is Maggie Haught. I'm on the leadership team at OpenAI. In particular overseeing the team that is bringing ChatGPT Enterprise to market. We launched about 10 days ago. Prior to this I spent two years at Webflow building out the sales team there and then prior to that six and a half years at Slack. From the very first founding sales hire out of headquarters, all the way through acquisition to Salesforce.

Incredible background. Well, I'm really excited to talk to you all today about the way that data and AI are changing the way that applications are built. It's just been incredible last six months. Look at the rapid pace of innovation within LLMs, how they're really transforming a lot of the different ways that people are using software and maybe Jordan, we can start with you at the data layer you built. You're commercializing DuckDB, one of the most popular databases now in the world. Super high performance in memory database. How are you seeing AI change the way that applications are built and the way that people are using data.

So to start out with, I was a little bit of like a resistor thinking, well, you know, this isn't really going to change our world, you know, maybe it's changing a lot of other things. But then I was talking to somebody who was, he was giving a course on data thinking at Princeton and he was teaching people how to ingest data, how to query their data and then how to visualize their data. But he wasn't teaching them any programming languages, he wasn't teaching them SQL and he wasn't using higher level tools that can automatically do this stuff for you. He was just using ChatGPT to generate Python code, to ingest the data, to generate SQL statements, to generate, to generate, then the Python visualization stuff. And I'm like, wow. And this was a few months ago and ChatGPT had just been out and you realize that yeah, this is gonna change who can do this stuff and also change the skills that people are gonna need and the tools that you're gonna need to build. So I think that's one side of things. And the other side of things is just the developers, people who are building things with that need access to data, are going to be changing how they access that data. Maybe the V1 is like the vector databases, but I don't even think we know necessarily what V2 is going to look like because OK, you get those vectors out, what do you do with them? How do you actually access, access your data stores and relate that to what you're getting out of the models. So I think there's a whole big area of unknowns and exciting possibilities there.

Yeah, I mean one of the things you touched on there that I think is really important is how much LLMs are democratizing technology. It's really much easier for someone who's maybe not be as sophisticated with SQL to suddenly start asking and answering questions of their data in ways that would have taken them 3, 6, 12 months to learn beforehand. It's an incredible phenomenon and it's being adopted broadly within the enterprise. Right Sharon? I mean you're seeing this, we're seeing this just kind of across the board. You look at the total number of customers that are starting to use Microsoft's OpenAI technology has 3x I think in the last quarter in the reporting. How are you seeing enterprises start to adopt some of these technologies and how are they manage it? There's a lot of technology here, it can be complex, there's tuning, there's lots of different components to it. What are you seeing in the market at Laminai?

Yeah, I think even just zooming back out at a technical level, I think actually public data is running out for all of what LLMs can take advantage of. So actually the next frontier for LLMs is in enterprises. And I believe the best LLMs for this next wave essentially will be enterprise LLMs. And the enterprises that I'm seeing that are at the forefront of this, they have started to adopt techniques around fine tuning their models to personalize models for every single user and every single customer of theirs where the data doesn't actually leak between those customers, but the model is highly personalized according to their access controls for data and just any kind of complexity around their data slices and use cases. So that's what I'm seeing and it's honestly really exciting to see this because this will further democratize all the possible use cases that can be out there, you know, far beyond what I can, I know, even imagine.

Yeah.

So, okay, so what we're starting to see, we're meeting lots, I'm a venture capitalist, we meet lots and lots of different businesses. They're starting to offer these software that's enabled with AI. And there's a question that we're sort of wrestling with in the ecosystem. Which is it the vendor who's responsible for actually optimizing or are there people inside of the business that are subject matter experts? Are you seeing any trends on which side is going to be responsible or mostly responsible for optimization?

Yeah, so there's definitely this tension between the two where AI expertise and you know, it might take a couple hundred top AI researchers to be able to train some of these models, such as at OpenAI. However, I believe that, and this is based on my experience training these models, it's actually the domain experts will be driving the best models out there. It won't be people like me who can actually do all the model training, et cetera. All that infrastructure will be built out such that more people can actually define the trajectory of these models. And as we saw with ChatGPT, with more people being able to use these models, they thought of way more creative, different use cases than like I in a silo with a few other AI researchers can think of. So I think the trend we're seeing is really, you know, the, the typical developer, the infrastructure developer, even the front end developer, all these developers are able to, who are able to access that data and be able to manipulate that data are the ones who can actually help develop the trajectory of these models. And do the fine tuning and actually build out this next generation of LLMs for the enterprise.

Yeah, that's a great segue, Maggie. You're in the center of the world. OpenAI basically catalyzed big chunk of this movement you recently launched, OpenAI for Enterprise. Can you tell us what was that like? What are you seeing in the market? What sort of revelations can you share?

Yeah, absolutely. So I will actually break this down into three things. I'll share with you all a bit about our journey to building ChatGPT Enterprise over the last few months. I'll also share with you some of the learnings and kind of use cases and what we learned from working with our design partners. And then finally I'll share a bit about what is coming, which we're really excited about. So I was hired on to OpenAI in 8April, really with the mission of building out the go to market team for ChatGPT Enterprise. We knew it was something that we were going to build and that was because when ChatGPT launched early last December, all of a sudden we started seeing millions and millions of knowledge workers using this tool every single day for their jobs. But then you have kind of the, the age old security privacy IT side of the house that is saying, whoa, we don't want our corporate data being put into these models and trained on these models. So we put our heads together and said let's build an enterprise version of this that has things like data training turned off. SSO, SCIM, I mean all the enterprise Y features, SOC2, CCPA, GDPR, all the acronyms. So when we set out to really launch this, what we decided to do was to select a pretty big variety of design partners. So for example, Amgen, PwC, Bain, Samsara, we decided to go as wide as we could with different industries and use cases, quite frankly, because we wanted to see how healthcare uses this tool, how does banking use this tool, how does consulting use this tool? So we worked really closely with them behind the scenes of course, under some pretty mega NDA lockdowns for the last few months to understand what is it that the market wants to to see for a V1 launch. Throughout this, there was a really big learning that came out to us and it was the importance of developing alongside your customers the idea behind really strong use cases, enablement training. I think when we started off we just kind of assumed, oh, everybody knows how to use ChatGPT. But then we realized everybody uses it in a different way that they've kind of created themselves. So we've been Thinking very deeply about how do we work with these design partners and now the world to develop very specific tailored use cases to their industries, to their Personas. ChatGPT can be used by every single vertical, every single company, every single department within a company. It's just mind blowing. So that was a really big learning for us, was really invest in customer success type work. Early on we set up Slack Connect channels, we asked these design partners to have weekly syncs, we sent out surveys to all of their beta users. And so we feel really good about what we launched last week. And we've also been quite public with our roadmap and what we're building here, which I think is also quite unique. Normally companies are a bit cagey and we've shared pretty broadly that the single biggest thing that we are working on right now is this idea of customization. So imagine being able to have your entire company's corpus of data or all of your own personal data infused in with ChatGPT Enterprise and basically being able to query, you know, all of my top sales deals, what were common trends across them and having that all come back for you. The other thing that we are also building right now, which is really exciting, is a business version, so more of a self serve version for SMBs. We realized, you know, we went out of the gates with enterprise, but a lot of times SMBs just don't quite need all the enterprise Y features that we launched. So there's just been pretty huge demand from SMBs, from startups, from founders to be able to adopt this. The use cases are endless Code interpreter, which is now renamed Advanced Data analytics is probably the most powerful tool I've ever seen in my life. You can essentially ingest it in hundreds of thousands of rows of data or spreadsheets. I think about it as a super calculator. I use it personally for looking at leads, looking at sales data analysis opportunities, win loss rates, everything. And it basically just synthesizes anything that would typically take someone a day to parse through in minutes and then down to the bare bones of recapping customer calls, feeding in transcripts from Gong and having it analyze that and then write in the next steps, follow up email that is like beautifully perfect, personalized and tailored. Which would have taken me half hour per email before I can now do in 10 seconds.

That's amazing. I think the point that you hit there, that solution sale where you have to take something that's a general purpose, real innovation and really solve an end customer problem, that's what we're seeing in the ecosystem today, it's like people, we're really excited about the technologies and industry, but we need to figure out, okay, what is the problem that the end user really needs to solve. Sharon Jordan, are you starting to see, are you focused on individual Personas when you think about taking your products to market or seeing different use cases arise?

I think there's the sort of developer Persona and I think those, I think the way those, I mean as you mentioned that everybody uses ChatGPT in a different way. So the developer is going to use it in a different way than an analyst is going to use it, which is different than a data scientist. And I think, yeah, there's going to have to be different tooling for each one of those.

We're hyper focused on the developer, so the software engineer, someone who can actually deploy a real app in production. And we're just really focused on getting these basically production use cases out. And we believe that the software engineer who sits most closely to their production apps will be the one to do that. And we've been finding that they're able to leverage data really effectively and efficiently. They're able to hook up their internal systems, whether that be recommendation systems or search systems, into these models as well. So they're very capable at pulling together all of these different APIs and continually improving these models on a schedule. And so we're hyper focused on this Persona. But I'm also really excited to see just so many people be able to use this technology.

Yeah, that's been a big change. I mean, 10 years ago, the way that I thought about the data stack was you had systems that produce data and then they would put it into like cloud data warehouses or data lakes. And then there were three consumers. There was the BI use case, exploratory data use case and then there was machine learning. You're trying to build data products. And now it really seems like those, those machine learning engineers or that functionality is becoming part of the core products, becoming part of the core workflow. Right. Like mag, you talked about gong like analyzing those sales calls that's part of the product. And so the machine learning engineers are now starting to be put into the path of production. And there's this blend, there's this mix between, okay, you've got a classic software engineer who's used to shipping code with like CICD and the classic workflow. And then you have machine learning experts and bringing in different technologies, right. Whether it's like data caching or customized models or off the shelf APIs. And you're putting it into production. It seems like there's a merger there of two Personas that historically have been similar, separate, and the landscape is sort of rising up to how do we enable those people to be as effective as they can be using things that maybe their expertise is sort of out of their domain. So you need a higher level of abstraction. Is that what you're seeing in market, Maggie?

Yeah, absolutely.

Yeah.

And I think that's incredibly important, actually merging those two together because historically, maybe to give a fun anecdote, one of my friends used to lead engineering at Uber. And so a thousand people engineering team, and what he found was the machine learning team, you know, was building models in Python notebooks that were obviously then had to be sent to the production team to then, you know, like ship it into your, into all of our apps. And one time, because Uber was, you know, moving very fast, let's put it that way, they had like just fired the person who ran this other model. So they had to literally go find the person's personal laptop, open up the Python notebook, rerun the data on these like a few states where they had to like have an updated model and then somehow rip that model out and thank God they found that laptop and put it into production and now it's shipped on our phones. And that is an insane process that's super brittle. And I guess another anecdote is just, you know, it's really hard to get some of these systems up. And in machine learning, you know, a lot of AI people are like, yeah, we push the performance of this model and we define performance as accuracy or accuracy along these general benchmarks. Maybe it's sixth grade science questions or something, but a production software team will look at that and say, that's not performance to me. I also care about the speed, the latency of this API call. You can't give me a better model and give me 100x slowdown of this model because I'm putting it in an autocomplete type of interface that's not going to work. And so I think merging these teams together, bringing them together is really important to actually shipping these models in production. And I think historically it's largely been this like R and D team with a really thin line to production and oftentimes failing that. And only, only very few teams could succeed at that before. But I think now many more teams will be able to succeed at that. And it's just really exciting for AI more broadly.

Yeah.

And I think as those teams are merging, there's this sort of interesting debate, which is what is the lingua franca of what language are people going to be writing in? Is it Python or is it English? And so as people start to be able to use just English, natural language to do some of the tasks that were in the other camp, I think that also can help accelerate that merger. Even if sort of maybe fully getting towards you. Describe your program as sort of like an English language paragraph and have that turn into a production system, maybe that's a little bit further, further off. But I think as you start to kind of peel off workloads, it can just help those teams merge.

That's the thing that's so exciting when you think about Microsoft Office, 330 million paying subs, whatever. Let's say a third of them use Excel. All those people want to be able to access some more sophisticated data within the organization. They want to analyze it. Like you said, historically they've really needed to learn either VBA or SQL language. Not everybody really knows. But now if they can start to speak English, you really have sort of the power of data now being exposed to, I don't know, 10x100x greater the population. It's pretty incredible. So just as we've seen, I mean, we're talking about on the technical side of an organization, you really see a big transformation in what's happening. Maggie, you're a sales leader. Like, how is, how are you seeing LLMs and these applications change the way that companies are starting to go to market? What's going on in the front of the house?

Yeah, I think it's best to start with a bit about how sales has changed over the last few years, because every year it seems to be there's a new step. And I think the short of it is that AI is going to fundamentally change how the sales role operates and looks. And it already has. We're already well into that. 2017, 2018, it was pretty easy to prospect into companies. You know, prospecting is kind of the bread and butter of any sales organization to generate opportunities and customers. Then all of a sudden you have these tools of sales loft and outreach which come out and just take prospecting mainstream. So then all of a sudden, sales reps are sitting there saying, what do I need to do next to get in front of buyers? The next step was personalization. So years ago, personalization wasn't even a thing. Last few years, it's been all about, how do I research this company, this customer, this person, did we go to the same school? What can we kind of connect on together. Well now LLMs have made it so research and personalization can be done in less than a minute. In less than a minute you can have ChatGPT or Jasper or any of these other tools that are out there right now. Go write an email that sounds very personalized and tailored but was actually written by an LLM. So now this kind of next era of sales and sales orgs has to really be thinking deeply about how to interact with the customer. No longer can you get on a call and ask somebody a million discovery questions. You have to show up to that call with a strong point of view with how your product is going to save this company money or solve their pains or whatever their mission is. And the beauty about LLMs is now they can help you do that research in advance. So back in the day when I was sales rep, I used to spend two to three hours before a big meeting, even more researching and prepping and practicing my discovery questions and coming up with a value hypothesis. Now that time can be significantly reduced to maybe 20 minutes max and you can literally have the LLM BE role playing with you of the questions that you are going to ask and the value statements that you're going to bring. And now I have, you know, LMS also on the other side of it are going to save so much time for the sales professional in terms of administrative tasks. No longer the days of writing, as I mentioned, long follow up emails or logging your notes into salesforce. You can literally feed, as we talked about, transcriptions from gong directly into there and use prompt engineering to write these beautiful summaries. There's all the new tools that are building slide decks for you, which is wild. So you're able to save on the administrative side of things. But the sales rep in this day and age and sales teams have to really step up with their value selling.

Do you think that changes the structure of a sales org at all or do you think we stay with like the SDRA model?

I think we stay with it for now. I mean I think AI, I think sales teams and companies that adopt AI are going to significantly outpace those ones that don't. So I actually think now like the common sales rep should be doing everything they can to actually learn prompt engineering, to learn how to leverage AI in their day to day to make them so much more efficient. I think maybe someday the model changes, but I don't think we're at that day just yet.

So we're all salespeople, right? If you're head of sales or you're A salesperson, you're a venture capitalist. We're all selling something. These tools, like you said, they open up like enrichment, discovery. How are you using LLMs in your day to day in order to improve your life? Are there tools that sort of stand out for you where you've seen a complete game changer? Anything kind of come to mind?

Are you asking me?

Yeah.

I think certainly people that are interacting with data, it's changing their lives. One of the things I do a lot of is I do a lot of slide decks and things like midjourney and generative AI just make it so much easier to have the right image. And yeah, maybe sometimes people have too many fingers or there's ducks without wings or kind of wonky eyes, but I think as the technology gets a little better and perhaps the operator gets a little better at prompt engineering, those kinds of quirks will go away.

Well, with mother duck, I mean, you've done an incredible job making that such an iconic part of your brand. I can imagine getting the duck anatomically correct. Really, really important.

It is important.

I think one thing that, you know, we were analyzing our data when it came to all our marketing blog posts or like tweets, everything going out, and there was a 10x increase in anything with an image. Right. And so it's just much more engaging for people to actually have impressions on. And so we found that just spending some time on midjourney was worth more than spending time on the blog post in any single way. Like, hands down, just like spend a few extra minutes here. Prompt engineering or really just like generating an image of your choice and clicking through them versus actually writing out the blog post that no one's clicking going to read. So I think that was a really big lift for us when it came to marketing specifically.

Yeah, just piling on those images are super helpful. I was unfortunately sued for using an image where I didn't have the rights. And so I learned a very important lesson. And so now most of the images on the blog post that I write are midjourney generated because now I can be confident that I'm not going to get myself into legal hot water again with rights issues.

Yeah, go more into that story. I'm kidding.

Well, so I mean, you know, I was like, I was, you want a hero image, right? Like you look through like the click through rates and the dwell times. And so I was using Google and there's a filter on Google where you can search for things that are Creative Commons license. And so I found an image on the Google Search results that ostensibly was licensed with Creative Commons. Turns out actually somebody owns the rights to the image, which I was notified later. And so now at least with like a synthetic image, you can be totally confident that effectively I don't know what the latest legislation is, but nobody owns the image or I at least own the image. So that's been a huge help. Do you use anything daily, Maggie? That really sort of where you see like a pretty meaningful improvement in your day to day?

I would say ChatGPT Enterprise, but that's not a fair one. I have been the last few years. There's just no tool that's better than Gong. GONG is a sales leader. GONG is getting ramped up as a sales rep or a new hire Gong for product. I mean as we've been building ChatGPT Enterprise, we would love to have our product leaders on every single call. It's just not realistic. So we can literally take snippets out of Gong and show share all the common feature requests we can drop in hashtags. I mean I'm on Gong's website, unlike from my previous company of speaking about them, but there is just no better tool out there than Gong. So I'd say ChatGPT Enterprise, Slack and Gong are the three tools that I use for many hours every day.

Yeah, that's awesome. We do the same thing when we meet founders. We record the conversation so we don't have to ask them to repeat. And we're starting to save all that information and produce LLMs on top that summarize diligence. And we produced our first investment memo summary from an LLM. And so we can just start to see as we gather more and more of this information, we can accelerate some of the things that are wrote within the org. The other thing that's been really useful for us is blog titles. So, you know, I think about like LLMs. I mean there's the incredible things where they're taking the sum total of like human knowledge that we've put on the Internet or the data sets that we've trained and then they can kind of replicate them. And you think about all of the amount of human effort that's gone into optimizing blog titles, that's all on the Internet, right? And it's all encoded within these LLMs. And so if I take like a blog post that I've written and then I ask one of these models to produce like three or four suggested blog post titles, I actually get three to four times better click through rates on the titles because I'M aggregating all that knowledge. So that's been, that's been a big, big help for me.

Awesome.

Yeah. Okay, so to bring us home, we'd love like, quick take. Where do you see for your particular company, like, where will we be in three years? How will LLMs have actually changed your business?

So I'm going to a little bit of a cop out on this one. And it was an answer that founder colleague of mine gave, which is before ChatGPT, he could sort of look at, look five years out into his business and be like, okay, I can see the road and I can see like, you know, where we're going and I'm pretty sure, like we're going to be successful. And then like ChatGPT comes along and like this cloud comes over the road and you just, you can't really see more than five feet out in front of you because, like, you know, as, as, you know, OpenAI may release something that, you know, makes something totally irrelevant or make, make, make it so that developers or analysts start to use some, some sort of other, other technology. I think the only thing that, you know, I think is clear is that people will, people will be doing things differently, their jobs will be different. I think, for example, like, the data space has sort of started to aggregate into these different job titles. Analytics engineer, like somebody who uses dbt, you know, data engineer, somebody who, you know, munges, data analysts, et cetera. But kind of as all of that stuff can be done with prompts, like, I think that there's going to be another. I think there's going to be a reshuffling of those roles.

I think every enterprise will have their own LLMs. And I think what that'll enable is in every single market there are domain experts, for example, like sales, they're domain experts. And what an LLM does is it becomes a scalable domain expert. And so for that market, there will be scalable domain experts for every single type of expertise, tailored to every single type of customer and user. And so I find that really, really exciting. So I think that'll dramatically transform job roles moving forward and that'll dramatically change how basically how businesses need to operate and who they need to even bring on and hire.

Maggie, in 60 seconds or less, predict the future for us.

I can't comment on OpenAI's roadmap, but what I can say and what something that I'm the most excited about in terms of an industry that is going to get, get completely upended is education. Just last week I was with the CIO of asu and he has these huge plans to bring AI tutor bots and AI coursework and really democratize the future of education so that anywhere in the world has equal and equitable access to education. And I think that's going to be one of the biggest impacts this world has ever seen. Is everything that's going to happen with AI and education, it's going to be just absolutely bonkers.

I could not agree with you more. I mean, I think the thing that we've noticed across the, across a bunch of different companies is if you have somebody who's new to a space, new to domain, and that could be math or calculus or biology. It could also be SQL or data engineering if you can give them access to an LLM, the quick feedback loops and a super patient instructor where you can say like I don't understand it or show me this or the other day I had written a line of Bash code and I had forgotten what it was like chat GPT, like what? What was it that I actually wrote in this piece of code and explains it to you? It's an incredible, incredible tool.

Yeah.

Thank you very much. Awesome. So we'll open up for Q and a here for 15 minutes. Any questions from the audience? Hi Maggie. I'd like to personalize my question first by expressing some appreciation for your 20 BC podcast. I learned a lot.

Thank you.

As a sales leader, if you're planning next year's sales plan, do you have any advice on improving an AE ramp time from let's say like six months to five months?

Absolutely. I think you can get ae. Obviously I don't know which company or industry, but I think five months is actually really long now for AE ramp time. I think there's a couple different things. Obviously I mentioned my love of gong, which is just getting your AES the previous existing calls in their hands asap. But there's also a lot of stuff that you can really encourage them to do with AI. So, you know, feeding data into an LLM or using ChatGPT and having them create quizzes, basically testing your product knowledge. A lot of stuff you can do with AI. A lot of just kind of traditional type things of certifications and whatnot. If you ever want to chat about it, feel free to reach out on LinkedIn. But there's so much that can be done to get AE's ramp time so much faster than it used to be. Appreciate it. Yeah.

Hello. Hi, my name's Lee Arthur from, from Blenheim Chalkot. A question for you Maggie, we think a bit about company market fit, not just product market fit. Especially as we're building young SaaS companies, we feel that they can easily be roadkill for bigger tech. And I think that ChatGPT and others enables any tech company to move into any of the space much easier. And I wondered if you had any advice in terms of competitive strategy for SAS companies who are going to market to enterprise, but Microsoft, Amazon, Everybody else, Enterprise, ChatGPT may come along and just add a feature on your business.

Yes, I think the biggest thing that a company can do right now is really differentiate themselves and really think about how you build your moat. I do see in the site, I also run a small fund with seven other women. And so we are seeing every single day all these startups coming across our desk. And where these startups, where many of them are going wrong, is they are building a thin wrapper around an LLM. And you need to do something such as sourcegraph, who is taking really rich data, sets of code and best practices and infusing it with an LLM. And that's really what's going to set up that competitive moat. Whereas my advice to all startups is don't just build a thin wrapper on an LLM because there's a great chance that a large Microsoft, Google, whomever might build something competitive to it.

Yeah, just to chime in there, what we're starting to see are startups that are chaining together LLMs in order to take much more complex workflows and simplify them, particularly across teams. And if you think about the last 10 years of software, a lot of the software that's been built is let's take a feature of a bigger product and just optimize that. And now what we're starting to see is no broader suite strategy where it's much harder to replace. And it's not just a thin wrapper. Great answer. Hi, thank you for doing this. The question is to you, Maggie. So I'm the founder of the startup called Talk to Data. So we are building code interpreter to work with not just a single CSA file, but across SQL databases and CSV files and stuff like that. So my question is, do you guys plan to release code interpreter API anytime soon?

And the second one is, so what.

Kind of integrations do you expect to see enterprises will be able to integrate into these complex data analysis tools going forward?

I unfortunately can't comment on OpenAI's roadmap. Whether we're doing it or not. I can't chat about that publicly. I'm sorry.

Okay.

Okay, thank you.

Hi Sharon, I got a question for you.

Here, here.

Oh, I didn't realize there's a third line. Yeah, thanks for speaking up. First of all, I mean, thank you very much for sharing. I actually just finished your fine tuning lessons last week.

Awesome lesson.

I learned a lot. But at the same day I actually read the news article from OpenAI that the enterprise is released. So it got me a little bit confused. What is the difference between alumni service and the enterprise account for OpenAI? Supposedly it can address your privacy issues. Enterprise privacy, data privacy issues for both of these solutions. So what's your ideal customer profile? Personas. Right. So I mean how, I mean as a company like us, right, so how do we choose which platform that we would like to go?

Yeah, that's such a great question. So the way we view data privacy, and this is really coming from our customers, is that they, you know, they want to own their data as they have for the past several decades. They also view LLMs as an IP as well. They also want to own the LLM which is really deeply intertwined with that data. And that includes, includes the weights and all. So we don't own the weights of the LLM at all. All we do is help manage the infrastructure for it. And so when it comes to data privacy, this means that you can even, you can choose to do this on prem, you can choose to do this on cloud, you can do this semi cloud, you can be flexible around it and you can build your models on top of that, where we'd retain zero ownership over it and you have complete control and personalizability on it. And that means when you fine tune these models and build these models further on your customer's data, this means that that can always be owned by you. That cannot be seen by us in any single way. Even if there's a leak or anything, nothing can make it so that we can touch that data or that LLM. So it's really about ownership and building out that moat. I know someone asked over there about thin wrappers. This is about building a much deeper moat where I think the future of those who, who will play in this space at all will be those who own their own LLMs. And that includes owning the weights of the model and everything so that you can actually manipulate them to do what you want the model to actually do.

One additional question about that is we do have developers working. I mean there are hogging face communities, there are open sources. How do we decide to go to Labni or build Something in house?

Yeah, I think that's a really good question. A lot of our customers are in this sweet spot where they have started to build out some of their own systems and using open source libraries just to understand how do I glue together all of this? When you're ready for more of a production environment and a production build of this system, then folks come to Laminai because you can scrape together, you know, hugging face, deepspeed, all of these different great libraries out there. But it might just work for one run or a couple research runs, but you can actually integrate it in a production system and stay up to date. For example, we rewrote the PEFT library because all it was doing was. And this is the. Sorry, the PEFT stands for parameter Efficient Fine Tuning. And the library out there is actually just, you know, the research group kind of just threw it up there. It's totally unoptimized for actual production systems. So we rewrote it and we just stay on the forefront of that and make it actually maintained for production use. And the idea is that this is our. Like we're going to optimize this, but we're not going to optimize anything when it comes to specifically your data domain and your market. And the LLM is completely owned by you, but the infrastructure layer is agnostic across different markets. I hope that's helpful.

Thank you.

That's helpful. Thank you.

Hi, I have a slightly. An extension to the last question and this is a common question to both.

You, Maggie and Sharon.

What kind of enterprises or verticals do.

You see building their own LLMs versus.

Using Chat GPT Enterprise version? Yeah, I just want to be. Because it's so confusing in a sense that last week we know we launched that we're building enterprise. So ChatGPT enterprise is really a tool for the internal employee. And yes, as I mentioned, we are absolutely building customization where you can bring in your company's corpus of data into that. But really ChatGPT Enterprise is an internal employee productivity tool. And then you've got our API side of the house where, you know, I think everyone's probably pretty familiar with those. While we do have an enterprise program for the APIs, I just want to make sure it's really clear of like ChatGPT Enterprise versus APIs are actually two totally different products.

Also our customers, a lot of them actually buy both just to be just very transparent about that. So they're often they buy both, they want to be able to use ChatGPT, especially internally. A lot of folks buy Laminai because they want to build these models into their products to have to basically be a market leader in AI for their market. So they're building these as more externally facing features. For example, one product, we work with a leading fitness company with about 6 million users and they are building out, you know, this fitness app and they want to be able to tailor this LLM to every single one of their millions of users without spending I think over a billion dollars in compute a month. So they're using us to be able to develop these models that are perfect personalized to every single one of those users on a huge swath of data from every single one of those users. So that's one example and that's an external facing one, not In I think ChatGPT Enterprise is internally facing specifically. And yeah, so that's maybe that helps, I'm not sure.

Thank you. Hello.

I have a question for Sharon as well. So I'm the founder of FI where we are making this kind of what called local first software. They are web applications that completely runs locally first. So as you mentioned there will be LLMs for every industry. As domain experts eventually when will think there will be even personal LLM that runs completely on your laptop. Do you think that's possible?

I think that's possible. I mean we already see great work on Llama Llama C C right out there that's running on these M1, M2 Macs. I think 100% there will be personalized LLMs that'll be either running locally or on cloud and that'll be hyper personalized to every single person. And I think that'll come from any enterprise because in any market you can imagine when you're shopping, let's say online shopping or something, there might be one person to not only your Persona but just you, very just you directly. Or there might be one for fitness. Like I said, there might be one. Someone was talking about writing SQL for example, so there might be one for each of those.

Yeah, yeah, yeah, that's what everyone would love to see. But to be honest, the open source model is by far not so mature as what we can get from ChatGPT. That's my personal feeling. So let alone that we are talking about fine tuning a personal one on the local devices, do you think how long that will happen in practice?

Yeah, so I think fine tuning we've actually found to be able to outperform the top LLMs open general purpose LLMs on specific use cases, not general purpose ones. So you can outperform GPT4 or Claude2 on specific cases that you want to actually do it in as long as you, you have data. And of course that is all these, these are still like hosted LLMs on GPUs. We also have seen this done on like on premise GPUs. I don't know. This I think gets a little bit closer to what you're saying. I think the, I think there is still work to be done between getting from those LLMs to the one that you want which might be like hosted on any kind of laptop. Is that correct?

It depends on how much computer by GPUs. Like all of the startups are running for H100 right? While the majority of public having as good As, I mean 2060 I guess.

Oh yes. I actually for fine tuning you actually don't need H1 hundreds to do a really good job and actually get peak performance on these models. I think that's a misnomer if you want to build out the whole foundation model that's you need tens of thousands of H1 hundreds but otherwise you don't need that at all.

Okay, thank you. Thank you. One more question. Thanks everybody. I have a question for you about the thin layer. I want to hear your take on how you do, how you pick early stages startups that they only have a thin layer about anything. What do you see? What do you look for in differentiation? Okay, so the question is if you're going to build a startup that is just a thin layer, how do you build a venture scale business from it? No more thinking about it as most of these early stages startups didn't have time to build something that thick. Everybody's thin anyway. Yeah, that makes sense. Okay, so one of the really interesting things about AI enabled startups is their growth trajectories are meaningfully faster than a classic software company. Like their time to 300k or 500k in ARR is much, much faster. If you look at the open source traction, the total number of stars that they're able to achieve on GitHub, we've never seen a 0 to 1 year 10,000 star open source project before. I don't know, maybe 10 this year just using LLM on LLMs. So assuming you've got a company that's hot, that's a thin wrapper. The big question is like okay, you're going to use that thin wrapper in order to do, in order to drive a ton of distribution. The goal is you're going to catalyze this audience, you're going to solve a very particular problem for that. And then at some point you have to be able to leverage that distribution across all the products or expand into workflows that are then more sustainable like our audience talked about. It's really like that's the goal. Anytime you've got a big technology discontinuity, all of a sudden distribution becomes much less expensive on a relative basis because people don't really know how to game the system yet. You can think about like the early days of Google or the early app store days, it was the startups that figured out the distribution, these LLM enabled applications. Everybody's curious and so as a result, people are really open to trying different things. And so if you can take that initial distribution from that wrapper and then use that to drive a bunch of distribution and create a mode, then I think you can build a sustainable business. Thank you. Thank you very much for the question. Awesome. Thank you everybody. Big round of applause for our incredible guests. Thank you so much for joining me today. So grateful.

Thank you.

Thank you very much everyone.

It.