Harnessing Traction As A Gen AI Startup | Samsung NEXT | AISF23

So this next panel, if it didn't give it away, our next panel is going to be a generative AI panel.

If y'all could please give it up for Sharif, founder of Lexica, Anil Mishra, co-founder of LangChain, and Hina, investor at Samsung Next. Hey everyone.

Hello everyone. Good morning to all of you. So I'm Hina Dixit. I'm an investor at Samsung Next. Samsung Next is venture capital arm of Samsung, and we invest in boldest founders out there. We have invested in companies like Stability AI, Mosaic ML, Tabnine, and many more in the AI sector area. We tend to focus in 6 segments. AI is one of them. Apart from AI, we focus on infrastructure, edge, robotics, health tech, fintech, and blockchain as well. I want to say that Samsung Next believes that startups are an indicator where the world is headed tomorrow. So we look forward to connecting with founders and empowering them and going from there. That's the mission that we come with. A little bit about me. So I've been with Samsung Next for around a year and a half, and I made investments in Stability AI, Mosaic ML as well. And before that, I was at Apple for around 8 years where I led various AI initiatives including like Core ML, Create ML Org, Xcode, iCloud, and many more. Today we are going to talk about generative AI, LLMs, and obviously like the hot topic, which is security and privacy. So before we dive in, let's have Sharif and Ankush introduce themselves.

Hey everyone, my name is Sharif. I'm the founder of Lexica. And just quick TL;DR on Lexica, it's about 8 or 9 months old right now. I started it when the Stable Diffusion weights were open sourced. So if Stability AI hadn't been funded, Lexica would not exist. So thank you. But yeah, we, we, I was using the Lexica or the Stable Diffusion Discord for like 4 or 5 hours a day. Honestly, I spent way too much time in there, but it was really fun. There were like millions of images being generated every couple of minutes, and I really wanted to just write better prompts so I could make better images because everyone was writing things like, oh, like portrait of a cute cat training in ArtStation Unreal Engine by Greg Rutkowski. And I wondered, do these actually make a difference? Like, are my images any better for writing like a full paragraph of tags at the end of my prompts? So I wanted a more scientific way to like create images. So I put the entire Discord, I scraped the entire Discord, put it all in the database, and that was pretty much the V1 of Lexica. It was launched the day the weights of Stable Diffusion were open source, and I think we timed it really well where we kind of rode the zeitgeist of Stable Diffusion. So oftentimes when people would share Stable Diffusion, they'd also share a link to Lexica, which is pretty cool. But yeah, that's Lexica.

Cool, awesome. I'm Ankush. I'm co-founder of LangChain, which is a company that's formed around the open source Python and TypeScript libraries. Our goal is very simple. It's to help developers build cool and useful LLM applications as easily as possible. So some more background on myself, I guess. I started my career at Facebook, then worked at Robust Intelligence, which is where I met Harrison, and then worked at another startup after that where I worked with Harrison's brother. And then, yeah, I joined Harrison Uh, to, to start LangChain. It's been really fun and exciting to see what all the developers are building, uh, with the framework.

That's great. Uh, shall we dive in? Um, so my first question to, uh, to you, Ankush, would be, what problems does LangChain solve?

Yeah, definitely. So, uh, as I mentioned earlier, uh, or just now, uh, our goal is very simple. It's to help developers build useful LLM applications. And so I would break it down a bit more and say that I think, or we think, that the most useful LLM applications are ones that are data-aware. So in that they allow users to hook in their own APIs, their own data sources into the LLM applications. The other focus is building agentic LLM applications. So basically, LM applications that leverage the model's ability to reason. So think of, you know, we offer ways to build chains in LangChain, and you could explicitly, you know, construct a sequence of steps. Whereas like a more agentic way of doing that would be using an agent where the agent is deciding which action and action input to utilize at a given step. So those are the 2 areas that we're focused on. I think the 2 value adds that we provide are a set of common abstractions and building blocks that allow developers to very easily, you know, write complex applications in a In very few lines of code. The second is a lot of chains, like pre-built chains and agents that you can use out of the box. And so we've seen a lot of developers using the pre-built, you know, like self-treat chains, like chat, you know, conversational retrieval chains, etc. But I think we're also like really excited about developers building their own, uh, chains and agent executors and stuff as well. So yeah, very cool.

So Sharif, my next question would be for you. How does Lexica differentiate itself from other competitors out there in creative tool space?

Yeah, that's a good question. Um, so right now Lexica is kind of a mix between a search engine for images and also a way to generate your own images. So what we found early on was people were searching for a lot of prompts and images, really as a means to create better-looking pictures. And what we noticed they would do is they would actually have another Stable Diffusion interface open, either in another tab or in the background. They would kind of like alt-tab back and forth between Lexica and Stable Diffusion. It was really interesting because I found myself doing the same thing. So we ultimately decided to just build generations directly into Lexica's website. So now when you find an image you like, you can actually just press a button it loads it in the generator. Now you can generate variations or modify that prompt. And one thing we're really focused on actually is not just generating images from scratch, but also allowing people to generate or create new images and edit them as well. So for example, if you have an image you really like and you want to use it for like a specific asset, you have to do a lot of stuff to like get a working image from like this raw model-generated images. So this involves things like upscaling, restoring parts of the image, like modifying it within painting. Um, so rather than just being like a single model that generates images, we're trying to build a whole suite of creative tools in a way.

That's very cool. I also want to know, like, when you're trying to build these products or you're trying to partner, how do you think about the UI and UX aspect? And also, like, in the past, like, I was reading somewhere, um, and you have constantly said that, like, you know, you want your UI and UX to be, like, not only helpful for the creators but also— hope everyone is fine out there. So how do you, uh, focus on that? Like, how do you achieve that?

Yeah, that's a good question. I think UI and UX is by far the most underrated part of building any product that uses AI. Um, like, it's commonly said that you could have made ChatGPT using the GPT-3 API Because the model was there. Technically, it wasn't there, but in a way, you could have theoretically made a ChatGPT product using the OpenAI API back in 6 or 7 months before GPT-3 was released. It was really the interface of chat that made it really accessible to a lot of people. Similarly, there are so many really great companies waiting to be built, stuck behind Gradio UIs and Colab notebooks, that all really need is a way to deploy those models to an inference API and a really good interface, and you have suddenly millions of people using that tool, which I think is by far the best way to think about building products is how many people are using this weird collab notebook that is kind of obscure but people really love, and how can I turn this into a product that actually makes it easy for anyone to use? I think that's by far the most underrated part of building products right now, especially with AI.

Yeah, I completely agree. For LangChain, we think about UX in a couple of different ways. One is developer UX. So we want to make sure that we have the right abstractions that make it very easy for developers to build LLM applications. So we want to make sure that everything is clean and easy to use and plugs in well with their custom integrations and things like that. And for that, we get a lot of great feedback from GitHub issues and Discord. So, you know, for example, like some people find our memory abstraction a bit confusing. And so we know we're doing some work there and like same thing with like chat models versus like instruct models. And yeah, those are just some, you know, examples in the past in terms of abstractions. But yeah, as Sharif was mentioning, like user-facing UI/UX is also very important.

Yeah.

And we want to— this is more important for developers that are building with LangChain because we want to make it easy for them to build nice UX, user experiences, right? And so some things that we've added in the library to enable that are like streaming and callbacks so you can get intermediate results because these things are are slow, especially if you're sending a lot of stuff to the LLM. And so just giving the user some feedback about what's going on and that the application is making progress, I think, is really important as well.

That's very cool. I was reading earlier that LangChain also supports building UX using documents and it supports, like, you know, building UI as well?

Yeah, so we don't actually have a UI library, but we do, like, you know, a common use case of LangChain is building chatbots over your own documents. And a lot of people who are way better at building UIs than we are have built some really cool, like, chatbot experiences. So the way this usually works is you need to load your documents from your from, you know, where they're stored. So it could be like Google Drive, it could be PDFs, et cetera. You need to load all of those documents, you need to chunk them up, um, and, uh, embed them, and then put them into some type of, like, vector store. Uh, there's a lot of vector store providers that we've integrated with. Um, and then, uh, basically, you, uh, during the query step, you extract relevant documents, And then there are a number of different mechanisms you can use to, like, put the extracted documents into the prompt of your LLM. And then the LLM has context over, you know, like, how to answer the question. And so, yeah, I think, like, a common problem is hallucination here, where the LLM is kind of, like, making stuff up. Are giving you answers that are not in your documents. And yeah, I mean, there are like, this is, this is a very new area. And so a lot of the techniques have, are sort of still in development. I would say that, yeah, I think one, one category of things you can do is like just basic prompt engineering. Like, I think in general, like telling the LLM politely not to make stuff up goes a long way. I think in addition, like, making it very clear, like, in the prompt, like, this is the document, like, only use this to answer your question, to answer the question, is important. Making sure that your retriever is working correctly and, you know, you've configured your document chunking correctly is important and experimenting with that because that makes a big difference. Additionally, I think LLMs are pretty good at critiquing. So adding another post-processing step where you have an LLM look at the result and the extracted documents and judge whether or not it was making stuff up can actually help, even though it's like another LLM call. So yeah.

That's very cool. So in the recent times, we we have come so far with the LLMs, and now like we have this ability to fine-tune these models to solve problems in various verticals and for various use cases. And not just that, the compute has become like, you know, somewhat cheaper as well. Obviously I invested in Mosaic, so they have been doing some great work with reducing the cost for compute, and during the— especially during the training process. Um, given like, you know, the research community has come so far and now like we are in the commercialization phase of AI, what are some of the unsolved problems in LLMs you both are thinking of like right now in terms of solving?

Um, yeah, so I think like you mentioned hallucinations as a big thing. A few other things that are kind of somewhat related are prompt injection, where you pretty much just ask the model to ignore the prompt and have it do whatever you want, which I think becomes even more dangerous when you have models connected to data sources like other people's emails and their contacts. Yeah, you're probably a better answer for this question because we mostly work with diffusion models, but we do use LLMs slightly, but not as much as you guys.

Yeah, definitely. I think there are honestly a ton of unsolved problems with LLMs. I think if I had to narrow the— if I had to give a more direct answer, I think that one, area that's caught traction recently is autonomous agents. Anyone who's used autonomous agents has probably realized that they're really cool and really powerful, but often it's— I'm not going to say easy, but it's more straightforward to cherry-pick examples and have them do cool stuff and post it on Twitter and get a lot of likes, but it's really hard to put these things in production for some of the problems that I mentioned earlier, like, you know, hallucination, not necessarily like hallucinating off, in this case, not hallucinating on things that aren't in your docs, but hallucinating on the action step, like picking tools that don't exist or picking like wrong inputs to those tools. That's kind of like a common problem with agents. And they're also like slow and they can use up a lot of tokens. So people really want to start using agents, I think. one relatively unexplored area is fine-tuning models on agent trajectories. We saw hints of this with ToolFormer, but I think it's still relatively unexplored, I want to say. So yeah, I think that's a cool area.

I will add one other thing. I think the most exciting thing right now in large language models is Getting them to run locally on your own hardware, especially on your own laptop or your own iPhone. So I have a funny story, actually. I was flying back from Hawaii to San Francisco, and the trajectory it takes is over the Pacific Ocean. So there's actually no Wi-Fi on the plane if you're flying from Hawaii to San Francisco. But I was, like, really curious about manta rays because I had seen them in the ocean. I was like, wow, these are so beautiful.

Yeah.

Like, how much do they weigh? Like, what do they eat? Where do they live? But there was no Wi-Fi on the plane. It turns out I had downloaded a local model on my iPhone. It was a Vicuña model quantized, I think, 4 bits, and it ran on my iPhone 13 Pro at around 6 tokens per second. So I pulled it out and I asked it right in the middle over the Pacific Ocean with like no Wi-Fi for hundreds of miles, like, tell me about manta rays. And it slowly decoded it like token by token, and it wrote like a pretty decent explanatory paragraph on manta rays. entirely on my iPhone with no internet connection. And this is crazy. This is like you have like the world's knowledge compressed in your pocket. And yeah, not many people realize that this is going to be a thing. And this is with like some hacky Apple ML Core stuff getting it to run. You can imagine if Apple gets like their own Tensor chip working and they have like their own hosted model on device. Yeah, it's going to be a pretty crazy future.

I completely agree.

Yeah.

I completely agree with that. Since we are talking about on-device and these smaller models, I wanna also chat a little bit about, since all these models are going to be somewhat smaller in case and mostly on-device, a lot of these future trainings might not happen on the cloud anymore, right? And then there's this particular aspect of data privacy and security. So what are your thoughts about moving the training to on-device and decentralized aspect of model training?

Yeah, I think that's a really interesting area. And right now, I think a lot of people are still kind of in the prototyping phase and trying to figure out how to incorporate LLMs into their own applications. And so a lot of people are using the same model providers and just calling APIs because they want to get something working. But I think when things start moving into production, things like privacy and latency, et cetera, are going to be more and more important. And so, yeah, I do sort of see that trend because of cost and latency and privacy, things moving more on-premise. I do think that could be a trend. But I also think it's moving so fast, and it is too early to really tell.

Yeah, I don't know if you guys saw this, but in like the latest iOS release, which hasn't been released yet, it's an update, they added a bunch of accessibility features. And one of the most interesting features was the ability to train your own voice for people who are deaf. So you can— or sorry, not deaf, for people who can't speak well. You can train your own voice on your iPhone and then have it— you can type a message and have it pretty much text-to-speech it. and use that as like how you communicate with other people on FaceTime or through voice notes. And I'm like fairly certain Apple's gonna be doing this entirely on device, which is pretty cool.

I agree. I also wanted to talk a little bit about like, you know, this issue about the privacy and security and copyright, right? Like because Lexica indexes so many images like, you know, in its database, and some of those images like, you know, could be used using the seed and the prompt, right? Like, that is part of that particular image. Now, in such cases, like, how do you ensure, like, the copyright, privacy, and these, like, security issues while building Lexica?

Yeah, this is a pretty much very hard and unsolved problem. There have been some approaches to this, like adding tags to, like, your robots.txt on your website so you can prevent search crawlers from indexing your webpage. There are a few other tags that are being introduced where you can prevent people from training models on your data, which I think is a pretty good first step. But yeah, I think it's really hard. And not many people even know that people's models are being trained on their data in the first place. So I think having people become more aware of it in the first place is probably a pretty good first step. In terms of data privacy and security, Lexica by default is pretty open. So anytime you generate an image, it's public and available for anyone to see. which kind of has this really interesting self-moderation feature because if you generate an image and you know that everyone in the world can see that you made it, you're probably not going to try to make things that are weird or creepy. Some people still do, but we try to stop them. But knowing your stuff's going to be public and linked to your name is probably like a pretty good first step at that.

Yeah, I totally agree. I think awareness is where you start, and then you start building like tools around that. I also want to talk a little bit— this is going to be my last question, and after that we'll be taking questions from the audience. But before we kind of like wrap it up, I want to like get your thoughts on the recent, like, you know, speech that Sam Altman gave in Congress regarding like— and he said this, that AI is going to birth twins, God and Satan. So I want to hear like your opinion as well, both of you, where AI is headed, what are some of the dangers out there, and how regulations would help us and how should we approach that.

Did he actually say that?

He said it like— that was the quote that I read.

Wow.

I was not there, but that's what I read.

That's pretty crazy. Yeah, AI safety is pretty important, and I think it's probably something we should take seriously. And it's also important to differentiate between superintelligent AI versus what we have today, which are really large language models. A lot of the times people are fighting different fights where they think that when people argue for AI safety, they're arguing specifically for models like GPT-4 to be pretty much moderated better, which isn't really the case. They're mostly talking about like, if you extrapolate the trajectory of progress upwards, what do those models way up ahead on the curve look like and what are they capable of? Whereas a lot of people today are kind of just still thinking about language models in terms of like, the safety debate.

Yeah, definitely. I think AI safety and alignment are super important. I think additionally, I wouldn't say this is as dangerous, but I think that there's such a fast-moving space and there's a lot of noise. And I think it's very, as I mentioned earlier, it's easy to sort of get caught up in the cool demos that are, which are also like super cool, right? But I think it's deceptively hard to get these things into production. And I think focusing too much on like, you know, cherry-picking examples and like, you know, just like building demos will, can sort of like hide that. sort of hardship of putting these in production. So I think a lot more effort needs to be put there.

I totally agree with both of you. We'll now take questions from the audience. Yes, please.

Hello, how's it going? So this is for Sharif. So are you in secret super duper talks with Midjourney about when they're releasing their developer API? Because I think if anyone would be, it might be you.

Unfortunately, not in any secret talks right now.

Yeah. Yeah. Okay. That's my question. I think everyone wants to know, maybe.

Yeah, we get a lot of requests for an API, but it's not something we're focused on right now. It's a pretty common thing. I'm sure someone's going to make a really great API soon though.

There's a lot of discussion about regulation and particularly around licensing for copyright and generative models. One way of looking at that is it really advantages incumbent players who have a lot of money to pay for licensing for large datasets. How do you feel about the role of startups in this kind of growing ecosystem, and is there a risk that regulation or licensing cost burdens could kind of crush that out?

Yeah, I mean, one way to look at it, this push for licensing is like regulatory capture where only larger players like OpenAI can actually afford to license large datasets. And I think OpenAI actually did license the Shutterstock dataset when they trained the original DALL-E 2 model. For something like $10 or $20 million. But then you have open source and open nonprofits like LAION coming along and pretty much recreating larger datasets by an order of magnitude compared to what they could have licensed. I think what's really exciting is the push for open datasets, not only just open models. The LLaMA dataset was also a huge thing, and that was later used to train Red Pajama, which is pretty cool. We trained a model on the Red Pajama dataset, which is a recreation of the LLaMA dataset. But I think we do need more open datasets, more so than open models. There's a really cool competition actually called DataComp where they fix the dataset to around like 12 billion images. And the goal is actually to train not— so the model is the same, it's a CLIP model, but the goal is to train the same model on different subsets of that dataset. So it's a data-only competition where you hold the model constant, you hold the training flops constant, and you try to figure out what mix of data actually results in the best model. It turns out, of the 12 or 13 billion images, you can train on a 1 billion subset and that model actually outperforms the model trained on all 12 billion images. I think data quality is also really important and not many people are paying attention to it. We should pay more attention to data generally.

Hello, my name is Pian Pian. I'm the founder of A-Motions. We're building a startup to help managers and teams learn soft skills like emotional intelligence and help them be better Leaders and people with AI coaching. We're particularly thinking about using LangChain to train our AI model, AI coach, because we have some videos made with certified coaches and psychologists. So I'm particularly interested in like your thoughts, both of you or 3 of you, like how we can make our AI coach more distinct and different from any other coaches. any other AI coaches. Right now we're based off OpenAI, but we're thinking about building our own model and see how we can use other LLMs to improve the AI coach as well. Thank you.

Yeah, that's, that's a really awesome use case. I think, yeah, the most important thing is I think like setting up some type of pipeline where you're able to evaluate like changes to, that you make to your application. So maybe, you know, you wanna switch like model providers, maybe you wanna switch your prompts and just like understanding how they affect your application is like really important. And yeah, I think I mean, are you planning to train your— you said you're planning to train your own models, right?

For now, we're using OpenAI, but maybe in the future we can train our own model. We're bootstrapping right now, so we need the funding to train our model. But for now, it's like using OpenAI could be combined with LangChain or other tools too. And we have our own data. We have our own videos made with certified coaches and psychologists.

Yeah. Yeah, so I think extracting content from your videos, making sure that you're— Interesting. Hello? Hello? All right, cool. Yeah, I think, yeah, making sure that the model receives the right data, making sure that you're able to tell when it's making stuff up, evaluating, these are all important things for building a system that uses in-context learning. And then, yeah, I think making the judgment on whether to train your own models or not is— yeah, I think it's hard right now. But yeah, just having a system to evaluate those changes is the most important thing, I would say.

Hello. I just thought it was worth Putting a quick clarification out. The point on OpenAI spawning God and Satan was actually a meme article. He never actually said that. I just thought it was worth clarifying that he didn't talk about apotheosis in his Congress speech.

I think I'm next. Hi, guys. Here. First of all, quick question to Sharif and then question to Ankush. The quick question is, Are you planning to put in the search parameters the 3D assets that are being created with other— not only with the Stable Diffusion, but all these VR and AR kind of assets that are being developed right now?

That's a good question. Good question. Are we going to do 3D assets on Lexica? Right now we're not. We're just mostly focused on 2D stuff. I think you can take 2D pretty far. Like you can do like GIFs and then videos. But yeah, 3D is pretty cool though.

Thanks. And a question to Ankush, a little bit more philosophical. First of all, I'm a big, big fan of what you guys are doing. I've been working with game developers. So my lab, we have a small fund and we're investing in games. If I would ask you, what is your vision to help specifically game developers with AI, with the things that you guys are doing? what would be your answer?

You said game developers?

Game developers, yes. For game developers on AI specifically.

Yeah, I think that's a very interesting question. I think a lot of the stuff that's happening with agents is really interesting, especially for games and movies and books and things like that, because if you just, have like 3 or 4 agents just talking to each other and giving them prompts like, you are this like character and you're talking to this person, and have them play like Dungeon Master or things like that. You can really— they can generate some really like interesting stuff. And I think you can leverage that to build games. Like you can use that as inspiration for scripts or games and things like that. Or you can even have some type of— I've seen people build online dungeon master games that leverage LLMs to create scripts and things like that. So, yeah.

Maybe take it to a more ambitious level.

Should I go next? So, building on the question from that lady back there, I'm working on this venture called Work Genie. Which is affordable on-demand exec coaching initially targeting entrepreneurs. So right now I'm delivering the entirety of my stuff over WhatsApp. So I have roughly 40 gigs of WhatsApp data, and what I'm interested in is training an LLM over that data to automate some of my things and help make my workflow easier. A friend of mine told me I should look at LangChain and extracting WhatsApp into that. Any thoughts on that and any other tools? I should look at? Looking at this data which lives in WhatsApp and training an LLM over it to make my life easier. Yeah, I mean, I think I would just start by building a prototype where you just do like in-context learning, right? Where you just like extract data from your like WhatsApp logs and like put the relevant data in the prompt of an LLM. So I would start with that and just get something working. And then have a way to evaluate the answers that you're getting from your system. And then as you make changes, you can sort of evaluate the quality of the answers that you're getting, maybe speed, maybe token usage cost. And then really the decision is when it comes to train your own model, are you still hitting the same quality metrics? Right? Like, is it saving you money? Is it faster? Are you getting better? You know, are you getting like the same or better quality? These are all things to look at. It's going to vary a lot, like case by case. But yeah, I think it's, yeah, it's a lot of people have bootstrapped, you know, similar types of applications with LangChain. And then when it comes to really like productionizing and making it the best user experience, like evaluation is really important to evaluate changes. Yeah.

Hi, over here. I'm looking into building a tool that can automate digital marketing, and one of the issues I'm tackling is creating variations of product images.

So I was wondering what kind of data and what kind of approach you take in terms of having a one or multiple product images for an e-commerce store.

Let's take a white shoe, for example, and how to train a model that can generate new variations of that which look so much like the real product. So just to clarify, do you mean you want to generate product photos? Okay, so for Shopify stores. Okay, so you have a photo and then you want to make the photo look better. So you like remove the background and then take the product from like this iPhone photo, make it look like it was taken in like a studio. Yeah, that's interesting. Um, you could probably get pretty far without training any models. I would start by just like using Segment Anything to like segment the image in the first place. And then once you have like a really good mask for the image, um, the hard part's going to be lighting actually, because you need the lighting to match up from the scene and the image. Um, so that's interesting. If you could somehow create like a 3D map of the object somehow and then relight it. There's a really cool tool called ClipDrop Relight where they actually create like a 3D map of an image and then you can move a light around in the image and it relights it. So I would start with like just first segmenting the image and then second trying to get lighting to match exactly what you want. And then just adding in a background should be pretty easy. Like it doesn't even need to be an AI-generated background, but you can also just like, just use Stable Diffusion. It should be decent enough for like Thank you so much, Sharif and Ankush.

It was a pleasure talking to you, and thank you for answering all the questions. That would be all. Thank you, everyone. Have a great day.

Thank you.