Unaligned #3: AI development monitoring

If you're working at a big, let's say, Fortune 1000 company, what do the people who are running these businesses and are just starting AI projects, right? What do they need to know? And what do they need to know about you?

I think the hardest question when we talk about enterprises is, what is my actual use case? And that, surprisingly, is not obvious. I think the number one thing that these cup faces ask themselves is, what use case am I actually going to be using this for? And does it actually make sense? We've seen a lot of enterprises like the Fortune 1000s build out these systems and they're still like beta testing and they don't know where to use it. They're excited to use it and they're building out entire chains to use it, but they don't know where to use it, which I think is hilarious.

This one is going to be a bit nerdier than the usual Scoville interview. We have Justin from Helicon, founder of Helicon. Justin, first of all, I always ask people, who are they? So who are you?

Great question, Robert. I am Justin. I am the founder and CEO of Helicon. Before Helicon, I'm just like a software engineer that stumbled my way into founding a company. Classic Silicon Valley story.

It's awesome. Yeah, you know, we, Irena and I went through 4,000 companies. We found 400 developer tools companies, right? People who are building AI tools for developers. And you're one of those companies. And then we went through a whole process and we ended up with three companies we wanted to talk to at first. And yours is one because you're building monitoring systems for developers. So tell us, first of all, before we get nerdy and start talking about rag and architecture and what's going on in the AI space, I want to know what are you doing for developers in the AI space?

Right now it's just monitoring. But the idea is that we eventually want to get to a point where if anyone uses Helicon, their app just gets better over time. So right now we're in phase zero, where we're just building out the guardrails and the monitoring and observability pieces in order to get to that point. And we're just trying to meet the market where it's at. And everyone's using OpenAI and Anthropic. So we built out systems to monitor OpenAI and Anthropic calls very easily. So that's where we started. And yeah, we're super excited to. We are also building like a ton of tooling around that. And we're super excited to hopefully work with our customers to make their apps better. And by making their apps better, reduce latency, reduce errors, improving like performance and quality.

I've had a tech blog for a long time and I've talked to monitoring companies that monitor the old world like Microsoft Windows, Windows architectures or whatnot. What makes this new world different for somebody like you? Building a monitoring tool. What's the architecture? That's just. Let's go into that. What makes this world different for a tools vendor like you?

Great question. I think the main thing that makes things different is that the outputs are non deterministic. So you actually now have these, these for the first time in history, we're making massive calls to these systems where you don't know exactly what's being outputted. You make a call where it's like, hello. And then you want to understand when does it fail and what is my like success rate. So that's something that is like a new challenge where normal observability tools will just be able to aggregate that stat and be like, oh yeah, it's very deterministic. Like we know what will happen if we throw a duck into the system. We're going to get a duck out or something like that.

To translate that to English. If you have a large language model, which is the engine that's running these new architectures, if you ask a question of these large language models, the answer that comes back is slightly different every time you. Even if you ask the exact same question, it'll give you a slightly different answer. Right. It's like an autonomous car going down the street. It goes down the street slightly different every single time. Right. It doesn't go down the exact same path. So it makes it hard to test or keep track of when these things are working. I'm talking to a lot of entrepreneurs and they're talking about some of their systems have several of these large language models talking with each other along with a bunch of smaller models. Now that that might be answering. This is a mess for a developer to keep track of, isn't it?

Exactly. That's exactly right. Yeah. And we have all these not only mentioning the different models that they're calling, but like sometimes they link to each other. So you'll have one output of one model be an input to another model. And people are developing these crazy complex systems.

My psychiatrist who built one of these systems had a problem. The AI guardrails. Okay, let's back up. My psychiatrist talks to people who have had Childhood sexual abuse and stuff like that. Right. So they're talking about a topic and all of a sudden the AI guardrail comes up and says, hey, we don't like to talk about that topic and fails. So this proves that there's a new failure that you have to monitor for and figure out what's going on in the system. Why did it stop working all of a sudden? Stop taking notes in this case. Right. Tell me about some of the problems you're seeing AI developers have to deal with that a Windows developer didn't have to deal with 10 years ago.

You're touching on something very important here, which is we are basically becoming very needy on these model providers like OpenAI, and we are just hoping that whatever new model they release isn't going to break our systems. And they are adding a lot of guardrails, which is good because it's, you know, they have an ethics board and all this stuff to hopefully make their systems better. But a lot of people, especially in the developer community, are having frustrations, like your psychiatrists, where they want to be able to ask these hard questions because they have. Because they have this need and they can't. They're stuck. And we're seeing this as well with quality. I'm getting a lot of reports saying that the quality is just going down because they are expecting it to output. Like for the past models, there's always output a certain way. And now they're adding these guardrails and now it's adding a lot more fluff to protect itself. And then now we're starting to get into some, like, dicey ethical questions.

Yeah, this is a new world. A lot of companies are trying to build these personal AIs where you're going to be talking to the AI. Even OpenAI is using just turned on these new voice features where you can use your voice to talk to ChatGPT and it uses a voice to talk back to you. The problem there is, as a human, when we're talking, we're often rapid fire and we can talk real quick. The AI chat GPT takes 3 to 12 seconds to answer back sometimes. Right. There's a latency. Now OpenAI has figured out how to put up a little short answer first before it gives you the longer answer. So there's some sort of architecture with a smaller model probably that's doing that early answer and then a bigger answer that gives you a more accurate answer. This is a new kind of architecture, isn't it? And it's a new kind of problem. How do you get latency out of this system, out of this architecture?

Yeah. Is your question around how do we track latency around the system? Yeah.

What's your role in this engineering effort to get latency out of the system? But then what are you seeing happen? Because I'm sure you're working with some of these developers who are trying to build these systems that'll give you a new personal AI or a new holodeck or a new kind of NPC in a video game that you something in a video game and you're talking to it. If it takes eight seconds to answer you back, it's not very satisfying. Right. It's for video game.

It's not. We're seeing a lot. We're seeing like a push to do Edge LLMs, so stuff that actually runs on device or on people's GPUs. And a lot of our customers are really ticked off at the OpenAI latency. So they're switching over to smaller models like you mentioned, because they're just, they can. They have more control over how snappy it can be because they want it to be snappy for specific use cases. So a lot of people are moving to specifically around the Llama 2 models and they're using Hugging Face released this like inference package where people can just deploy it on whatever like AWS bedrock and stuff like that.

Where I was going is things that don't get measured don't get improved. Right. And you're a tool that's measuring and watch monitoring the health of all these systems going on. Your tool is probably going to be a powerful weapon to try to beat the, to get lower latency in these systems. Because soon these systems are going to be orchestras of AIs. I know people are building systems with a bunch of AIs working now. You have to monitor everything to make sure that one component isn't sitting there taking 20 seconds to answer.

Yeah, exactly. And making sure that we can relate things. So we helicon, we released this thing called Jobs, which is essentially talking about what you're mentioning, where you have these systems with multiple LLM calls that are like very complex orchestrations. And we built out this new feature actually just got released like Monday, where we are actually tracking Jobs, which is one inception point and then it branches out to multiple calls that all run in parallel or all run sort of. One job might depend on one another, but another job will output like 10 jobs or maybe have 10 dependents. There are these like complex web of instructions that we are now like visualizing and representing and like giving feedback on. So maybe the user might give feedback on step 10, but maybe that's because step one failed, stuff like that.

Yeah. You've mentioned that developers are trying to push to the edge. Right. I'm talking to you on a Macintosh. And in The Macintosh, the M1 processor or M2 processor, 21% of that processor is for neural, for AI. Right. So it's like a machine that's just sitting there unused on my machine. It's not doing right there, even though I'm running rewind listening to this. I'm writing a transcript out of this interview. Right.

Yeah. Robert, I have a. I have the hottest take. I think it's not even a hot take, but I've been saying this for like, for like years. I, I think Apple is going to do the biggest mic drop on LLMs. I think they're going to release an LLM that's just going to blow away anyone else's like that, like works on the edge. I think that they're waiting for it.

They're working. They're behind in some ways, but ahead in others. They certainly have this unused inventory of semiconductor. Right. Huge amount of computing sitting at the edge that's not yet being used. So your customers who are building all these new AI systems are going to have to build a hybrid approach. Maybe run a lot of things on the phone or on the computer user, but maybe have a bigger model running Nvidia cluster up in a cloud somewhere as well.

I think it depends on the use case. So you're going to have those products where it's like an AI chatbot. It's a game. Those are going to run on the edge where the stakes are not as high. I think you'll always have the models that take like hundreds of millions of dollars to train, that can't fit on your computer. It has to fit on an H100. And I think we're always going to have that sort of dichotomy where it's like you have these tiny models for these use cases. But for any serious tasks, like anything that like Hudson river is using or anything like Jane Street's using, they're not going to want to use your M1 processor. They're going to be using the big.

H101S Nvidia card up in the data center.

Yeah, exactly.

Basically what are. Let's talk to the business people before. Before we get really nerdy and dive in for development.

Yeah, yeah.

If you're working at a big, let's say Fortune 1000 company. What do the people who are running these businesses and are just starting AI projects, right? What do they need to know and what do they need to know about you?

I think the hardest question when we're talking to enterprises is what is my actual use case? Like, where can we inject this into our systems? And that, surprisingly, is not obvious. I think the most obvious ones is replacing any traditional NLP workflow, like document parsing with these language models. But I think the number one thing that these companies need to ask themselves is what use case am I actually going to be using this for? And does it actually make sense? We've seen a lot of enterprises like the Fortune 1000s build out these systems and they don't know, like, they're still like beta testing. They don't know where to use it, they're excited to use it and they're building out entire teams to use it, but they don't know where to use it, which I think is hilarious. So I think that's number one. Number two, I guess, like on the stack side of things, like you definitely want to. The one thing that we, what we saw when we were like iterating through products and we talked to Copy AI and we talked to Notion as well, in the early days, the first thing that they built out was Helicon. They built it out internally and what they really wanted to ask themselves was, hey, are my users actually enjoying this feature? And is it actually outputting useful things? And that sort of question is not something a normal dashboarding tool can just ask you. So that, that's literally the reason why you build telecone. And then, yeah, I think that's what you need to know. You should really. It's so classic. It's like the Paul Graham advice, which is just build something people want, make something you want, listen to your customers. It's YC advice. But yeah, that's the hardest thing we're.

Seeing for people who don't know Helicon. What kinds of things are you monitoring? If the first thing I do is buy Helicon to watch my systems, what kinds of things are you monitoring? And what kinds of things should I be using Helicon to monitor?

Yeah, See, what's funny is we monitor so many things at this point. We monitor our latency, we monitor different properties, we monitor like your different models. We have any different types of models. You monitor cost. We do like cost estimation. But the number one thing that users actually really care about, which is fascinating, and we like go through and make sure we Log all the user sessions, like track, like where the hottest areas are. Users just go into their request page and they love looking at their users. So they love looking at what their users are doing. And all they. And they just go through and just read every single one of their user requests. And we provide nice UI to do that. And that's the number one use case right now, which is so fascinating. It's. Hey, we built out all this stuff for you. Don't you want to use this? No, we just, they literally all we want to do is just go through and read their. Read what their users are doing.

Is that an artifact? Because AI is so new and these companies are trying to learn what their customers are really trying to do with these early systems.

Right.

They don't really know what the user really is going to do with these systems. And so they're very fascinated what the user is. How is the user talking a chat GPT or whatever. Right. What are they doing with it? Because we have a new machine.

Yeah. All of a sudden. Yeah, that's a really good point. I think you're exactly right, people. It's new, it's fun, it's exciting. And people are like, oh, let's play with the new toy. I hope it sticks, though. I think when we talked with posthog, which is a very similar analytics tool maybe a couple years back, open source as well, their main feature is the session live recording playbacks. And we're almost basically providing session live recordings, but just to your LLM model. And we're hoping that that feature is addicting enough for our users to keep coming back.

You mentioned cost monitoring. If you're building a system with 40 LLMs all answering questions, fighting with each other, debating the answers, rating each other's answers, how if you're using ChatGPT's API to do that, how are you paying? And then how are you monitoring that cost? And then how are you monitoring the cost if you have a bunch of LLMs all arguing each other out, which might not be an API charge, but might be a processing charge on your phone. Right. If you're wearing out the phone, you got to monitor that too, Right.

So for. It's basically like a giant function that we throw in. We built it out. We like hand roll every single cost calculation for each model. And a lot of these models, like for anthropic and for OpenAI, they don't make our jobs easy. Like they don't give you how many tokens it was, or they don't give you the format or the price per token easily in a nice like queryable way. So it's a lot of manual work to get this all to behave correctly for the models that don't exist, like you mentioned in the cpu, the CPU way for a model that isn't on a platform that you want to use. On our platform you can, we have this custom model support. You provide a cost estimation pretty much on. And we like can help you make that cost estimation based on like previous traffic. And it's just like an estimated cost. It's not a known cost.

It's cool. In five years, what do you think people are going to be using monitoring for once they get over this early phase of just trying to figure out what the user wants. But in five years they're going to know what the user wants and they're going to have very complex systems that are running, serving maybe billions of users possibly. What are they going to use monitoring in that age? What does monitoring look like in five years?

I think one things will always stay constant. People are always going to want to know their user behavior. People are always going to want to monitor how their users are doing. That's consistent. As far as the more like juicy things, I think one area we're seeing a huge influx of companies building in is agents. So monitoring agents giving, if you're familiar with like OpenTelemetry graphs, giving hotel graphs around these agent workflows and making sure that you can have like full session replay, not just on the single call itself, but be able to get full session replay on the full agent. So you can say, hey, I want to capture the agent in this state and I want to be able to play it. If this small thing was tuned, I think that is where that is a very hard challenge because these agent systems are so complex these days and there's so many different moving pieces. How do you actually build something like that? I think the person that does that is really going to be wildly successful in 10 years.

Do you eat your own dog food? Do you use the monitoring tool to monitor your own?

All the time. I have a ton of personal projects and a ton of hack. We went through the scale hackathon. We won third place for our AI rap battle. AI rapbattle.com. you can try it. But it's. You put in two. Yeah, you put into famous people, you probably put in yourself and someone else and then get them to rap each other and they'll generate like an image with like sunglasses and they'll do the whole rap battle. But that was just like A hackathon project. We built it in a day. And that that is actually still maintaining like a lot of traffic. So we're still like looking at that. Within Helicon, we as a team, we try to do one hackathon project at least like a month. So like we'll all get together and we'll just build something kind of fun. So the last one we built was a movie pairing recommendation system and that was pretty fun. We figured out like, hey, I like Tinder, right swipe right, left swipe on movies and then you find someone to pair with. And then we've built out so many things. Like my personal website, I'm dog fooding on Helicon as well. I have a little chatbot that emulates me. Yeah, we're constantly dogfooding, even within features inside hello cone itself. We have two, two features I call LLMs that are dogfooding. One, we have our playground where people can send requests and like test out what is. Let's say if they had a request that's sent to OpenAI and they want to say, I want to test it if it did this output. We actually do have that at the most fine grain level. Not at the agent level yet, but we have that at the most fine grain level. And it's basically called playground. And all that traffic is monitored within Helicon. And now we're adding a new feature in order to support more models in a more verbose way. We're building out something called this Rosetta for Helicon, which is basically using LLMs to automatically map these new LLMs. It's entendre. And yeah, we're dogfooding Helicone for that as well.

Very cool. Since you. One of the reasons I wanted the first developer company on my show to be you is because you're seeing the new architecture from the inside. And the fact that you're doing hackathons and coming up with all sorts of ideas is point on that. What are you seeing from the developer tool stack? Because that's even in the last year, it's changed a lot. What developers were talking about using seven months ago is different than what they're talking about using today. So tell me what the state of the art of development tool stack is.

It's funny because I would say the state of the art was LangChain and then LangChain fell off a little bit. If you look at the graphs, you can look at their install graphs. It's going up into the right. It has not fallen off by Any means people are still using it and it's getting more users every day for sure. But as far as what people are talking about, it seemed like everyone was talking about LangChain for a long time. People were also talking about guardrails and a bunch of other different types of like tools to help them build out these systems. Nowadays a lot of people are moving towards building out their own agent workflows and what we're seeing is that there's a ton of like we have, we saw this like auto GPT and like baby AGI and those different projects that kind of exploded and then also like top topped off. I think there is, I. There's not like an agreed upon framework that a lot of people are using. A lot of people are just like hand rolling a lot of their own tools right now. A lot of people use LangChain and we're burnt because they couldn't. When we talk to our customers we're like, they're like, oh yeah, we're using LangChain. And then as we keep talking, oh yeah, we're not using LangChain anymore. I'm like, oh, why not? Oh, because they don't expose this one variable we need from like the weaviate API. And they're like, we found it just way easier just to call the APIs ourselves and combine it and orchestrate it and just gives them more control. So I think like the problem with a lot of the tools that we faced that we saw was just like as soon as you wanted to get more complex, the systems failed. Like you couldn't do anything. Yeah, I wish I had a better answer for what tools people are using these days. I'm thinking about all our customers and they all just hand rolled all their own stuff. Right now they're just calling OpenAI anthropic.

And it's interesting. There's 400 companies chasing developers, trying to get them to use their tools. Right. And this whole process is so early that, that developers haven't yet found a company that really provides like you said, LangChain. I keep hearing the same things from developers that they're really struggling with using LangChain to what they need. Right.

You know what I think it is? I think it's back in the early 2000s when everyone didn't want to do code. So then WordPress came out and there was like all these other like no code tools. That's what I'm. It has the same feeling as that where all these different platforms that promise that you can plug and play all These providers together and build your app in seconds. It has the same feeling as those no cool, no code web builders. And it took 20 years for Squarespace to like, like in that race to become like dominant. Right. So I think it's still early days. I think people don't know what they want from this. These types of tools and what will end up winning on top is just those just code. Because you just have so much more flexibility.

You give me a little taste of the fundamentals of your company. How many people are working there, how did you get funded? How did you get started? Give me a taste what Helicon is.

So right now we're four. We have two employees and two co founders and then we also have intern that works every once in a while. We got funding through Y Combinator so we were in y Combinator's win 23 batch and then outside of Y Combinator we secured more funding from different angels and got some large checks from other investors as well. And right now we are hiring so we're looking to bring on one or two more. Sorry, one or two more engineers and we're just going at it. Yeah, I'm also. Do you have any other questions on that? I'm happy to talk about like our traction as well.

Anything like that would love to know. Yeah, absolutely. What are you seeing happen in the marketplace? Because this is such a fresh marketplace that people like are signalers of the health of the marketplace, I think. Right?

Yeah. So still growing. I think the market is still growing for sure. Like we're seeing a handful of larger companies start to reach out to us and say, hey, can you like solve our monitoring solutions? I think one reason they reach out to us is because we're open source. So a lot of the security concerns are alleviated because they can just self deploy. So luckily we can see what the enterprise traction looks like in the market and it's still pretty healthy. I think the main things kind of like touching on what we've touched on before is the enterprises are still struggling on what those use cases are, how it fits inside their product. But the actual charge to get there is very healthy right now. Like a lot of the folks we're talking to are hiring teams just to do this, just to figure out like, hey, let's build out our own internal tooling. Let's like spearhead this problem and let's like roll this out to all our developers within our org. And so that that's the enterprise traction on the startup traction. It's nuts. Like YC is Investing in it. They had a special AI interview cycle which you're seems like you knew about already. As far as like venture capital goes, they're pouring a ton of money into different AI startups or startups that call OpenAI even like these OpenAI wrappers. And we're not seeing those slow down, we're seeing those like definitely catch up and not even be like a winner take all market. So two of our customers are different YC companies. Like one is onboard AI, another one is like sweep.dev. these are both like have pretty large user bases and are in you would like from the outside it seems like they're competitors but they found different market shares that they can capture. And I think that it's still going to be the case. I think this market's going to be massive. And as far as our traction goes, like we're growing every day. We're growing at like, like a handful of new users every day and new paid users every week. And right now I think last I checked we're like for as far as like official companies using us, we're almost at 5, 500 companies using us. So we're like pretty stoked.

That's amazing. Already you're.

Thank you.

When did you start and why did you pick this? Why did you pick this? Out of all the things you could do, why did you pick monitoring?

So we were building something called Tabletalk, which was this thing, it's TableTalk AI, the website's still up. It was this thing that would let you talk to your database, you would plug in your database, it would personify your database you talk to like a human and it would be like a chat. And we built that DaVinci3 and then what happened was we were very cautious about the token window size. And then as soon as ChatGPT came out we had 8k token windows. I was like, oh this, this unlocks so much. And we were doing like all this like prompt stuff to get the format to work correctly. And then after, after a couple months we realized that hey, ChatGPT, like GPT4 is going to come out soon. We didn't even know functions were coming out but like that would have been annoying. But like the market itself is moving so quickly where if you're building a tool that depends on these models and you're building like you're building on an entire infrastructure that works well with an instruct model. But then next thing you know like a functions model comes out that will just blow away your existing architecture that was that's like scary to us. And honestly like that, in my opinion, that should be scary to anybody that's building one of these apps. I think one of the things that you want to do as a founder and as a startup person is you want to try to build for the future. And not knowing what the future looked like was scary for us. And we were like, okay, what's something that's consistent that we know for sure will never change? And it's monitoring and observability no matter what happens, no matter what hype cycle comes through. In the last hype cycle, the crypto hype cycle, one of my favorite companies was the graph, which is like this monitoring layer on top of these, like these chains. And I just had a lot of adoration for these like infra tools that as a developer I enjoyed like a ton using and I just wanted to build something that I would enjoy using.

That's awesome. Great story of starting a company. What should let's talk nerdy for the developers. For a developer who's starting down this path, let's talk to new developer who's new to AI. What do they need to know about this new world and how do they use you to build a great system?

Yeah, so I think the main thing, if you're just getting started, I think the main thing to learn is understand how the models respond and understand how to get it to respond in certain ways. I think as a developer you train this muscle on how to ask computers questions and how to be very descriptive with your problems. And if you work on engineering teams and you're able to communicate these problems effectively, I think you'll be a great prompt engineer. And I think that's like a very serious topic of understanding how to correctly ask the model questions and how to get the correct responses out of the models. And I mentioned the Functions API as well. Having a good handle on how that Functions API works can make or break how you use these models. Really make sure you're just like on the cutting edge of whatever OpenAI releases. OpenAI is definitely ahead of everybody else in the race right now. That's not going to, but we're not sure if that's going to be the case in a year. But as of right now, you should just look at like the latest OpenAI models, get a handle for it and just like start, just start calling it and use like whatever postman or curl API that you want and just start hitting OpenAI and play around with it. Try to get to output some certain thing and Then like any project, like any skill or software you're trying to learn, I always suggest to pick a project to build. Copy one of our projects in the past, build AI rap battles, or build a chatbot that personifies you, or build like a movie recommendation system, whatever you want. And then I think if you want to get started, I should think LangChain is a really good solution to at least understand how this stuff works. But if you're trying to build a company, I would say just skip the LangChain step. Just go straight into doing everything from scratch because you're probably going to end up wanting to do that. And then use Helicon, because Helicon will help you monitor all of those calls. Especially like once you are debugging. One of the problems with one of the problems our users feel is when they're using LangChain, they don't know what actually gets sent to the model because it's so abstracted away. Helicon helps you understand what's actually sent to that model. We have integrations with LangChain that works out well. And then we also monitor production traffic.

It's a new world. Do we have to treat multimodal systems differently than text based systems and, or do these new multimodal models that are just starting to come right now, are they presenting new challenges for you or for developers?

Absolutely. I think the, it's a huge. When we talk to like VCs, that's one of our moats. Because when you're talking about Datadog or you're talking about these other monitoring solutions, they're not going to build a visualizer for pictures. They're just not also. Yeah, just for people that don't know what multimodal means. It means that like you can insert pictures alongside what your prompt or you can output pictures or not just pictures, like audio multimodals, like. Yeah. And any of those types of formats. And I think for Holicon that is what we're aiming for. We want to be that platform. Right now it's just text, so it's pretty straightforward and it's pretty easy. It looks like our platform right now looks like a Datadog, but as these systems become more multimodal, we're super excited. Like we are going to build out these systems to help you visualize what was being sent over, help you see how different images respond in different ways or what like the combination different prompts and images and let you test that within our platform. And so yeah, for us, like selfishly, I'll talk about that first, I think it's, it's super exciting and we're like super excited to build out like cool visualizations for it.

I would expect the apps that are going to come soon are going to be quite different. I use Chat GPT's multimodal to take a picture of my spice rack and said, please take an inventory of my spice rack. And it did. Right now. Did it do it perfectly? No. So I need monitoring. To monitor when things aren't being seen or ingested or tagged, right?

Yeah, exactly.

The user gets mad.

Right?

Didn't do the right thing.

Yeah. And that's exactly right. And even like automated tagging, you might have these like robotic systems that like can pull in and detect plastics and stuff like that. Maybe hello can be installed in those systems and can verify. Like maybe you have a manual auditing process that you can just go through and like thumbs up, thumbs down, whether or not it was good. Like another thing we're building on Helicon is labeling and evals. That's, that's something super exciting. It's a new world. It's a new world. I'm so excited.

Can we talk to Helicon in English and ask it to show us the, you know, tell, tell us to tell us, show us new patterns in our 100. If you're monitoring, you know, millions of users, a little hard to see all that feedback in one place, right?

100%. You're talking about something that's super exciting for us is like how, like how do we add LLMs into our product? It'd be so dope if you could just like be like, have a chatbot to understand your LLMs. Like kind of mind bending, but yeah, like, why not, right? We have APIs, like a GraphQL API. Why can't we just have an LLM chat that automatically queries that API for you and asks questions on your behalf? That'd be so awesome.

Are you noticing that the teams that are building various different kinds of systems, like an autonomous car team, is probably different than an augmented reality team at Apple building for the Vision Pro. Are you seeing different kinds of apps being built and does that affect the kind of monitoring you're doing behind the scenes?

That's a great question. Yes, we are seeing a ton of different apps. We talked about agents before. One of our, I'll give one of our companies a shout out or one of our cup, oh my gosh. One of our customers a shout out, which is like Pluto hq. They're finance copilot. Finance copilot you can ask it what you think you should invest in. It's not, not financial advice. Use them at your own risk. But one of the things that they do is they have these crazy agent workflows and they want to be able to use hello tone jobs. And that is something totally different than someone that's building a normal chat bar like other companies that are on us just want to they will build a chatbot that resembles another character. For example, let's say they wanted to represent like Spider Man. That's like a very classic one. And those monitoring needs are totally different. You're not going to want that complete workflow agent flow view where you have these chains of thoughts and multi multiple calls. It's just one call.

Is monitoring important to build a system that improves over time? And how would I absolutely back your. What you're gathering into? Let's say yeah, my model building process so that my model next year gets better because of your monitoring.

Right. So there's two types of feedback and we're going to be adding a lot more. But there's two types of feedback. There's like implicit feedback and explicit feedback. So implicit feedback is if you're building GitHub Copilot and they have like code suggestions, you hit tab to accept the code suggestion. That is like implicit feedback. So that's like good. There's also implicit like bad feedback which is if someone has a completion and then they hit the back key or they delete it or they never save it, that's like bad. So Helicon has APIs where you can like record that for different requests. And then the explicit one where on chatgpt you like the thumbs up, thumbs down. Yeah. So you can also record that within Helicon. But yeah, you're touching on. So like right now we're on like phase zero, which is just build out, like just get the data in our systems. And phase one, which is where we're touching on is, okay, how do we now that we have the data in our systems, how do we build these evaluation pipelines to make sure that it is behaving correctly and that it's moving the needle in the right direction when we start to build out more in phase two, which is we already have caching to reduce latency, but we want to build out a lot of other tools to like help prompt quality and lower latency.

Are you building new systems that ask the user for feedback? Because in my Tesla car, if it, if it terrifies me right, it tries to drive oncoming traffic or Run a red light or something like that. Terrifying. Step on the brake, I take over and then I start talking to the car, right. And telling it, hey, you just tried to run a red light. I wasn't safe. Right. There's a system at Tesla that's ingesting all of that and using it to prioritize the engineering team's workloads and build systems that get better over time.

We are in a way, we are in a way trying to listen to that feedback. If you scream at your car, we'll hopefully capture that. I think I, I really like that example of yeah, we're, we're not gonna, I, I think maybe this is obvious or not, but we're not actually gonna be building systems that would inject in people's products. It'd be more like the users like Tesla would have to build something that like calls us. But that's exactly right. If you're screaming your car because it almost ran over somebody, we want to be able to capture that feedback.

It's a new world and there's a lot of new workloads coming. Right. There's robots coming, there's augmented reality glasses coming that we're gonna.

I know, I just, I got retta ones. I was like, oh my gosh.

Yeah, you know, we're. You're seeing the development world from the inside and I, I keep seeing all of these new models, Mistral and Llama. And what are you seeing happening in the model space? Because there's dozens now of models.

I know, it's crazy.

There you go. It is crazy. And what, how are you thinking through this rapidly changing place where there' there's new model every week and, or now you've even once you get a model, you start fine tuning it and changing it for your corporation.

Right?

Tell me a little bit, what, what are you seeing happening in the model world? Give me a state of the model.

World and how that the state of the model world for sure open AI is still on top. I think OpenAI is on top until people want security or speed and then the next one people go to llama. People might use anthropic for their larger context window. I think that will be short lived. We'll probably see OpenAI come through and build something with a larger context window as well. As far as like quality goes, a lot of our users say OpenAI has like the best quality even compared to the Llama models. And so that's like the market that's like really the top dogs right now. And then as far as like a ton of different models coming out. I think the, the greatest illusion that I draw on is, feels like databases like when Mongo and Redshift and all those started coming out and Snowflake and whatnot, like 10 years ago, I guess more than that now. But it feels like that we're, we're going to have so many different databases and they all live, they all, they're all here for their own different use cases. And I think that's what's going to happen. You're just going to have a hundred different models that are all going to be different. Like you're going to have some website that you're going to go on. It's going to be like a B comparison, different models and you're going to see them all. But you might want to use even a helicon. We use two different databases at most companies when they start to scale, they use like 10 different databases. I think that's like the greatest solution that I can draw to is. I think you're going to use, you're going to start out with, I don't know, like the postgres of models is probably OpenAI or Llama. Everyone starts with postgres. And then as soon as something doesn't scale well or you need to do XYZ thing, you're going to move and use like an OLAP database. Use like Redshift or something and. Yeah, and then you just start like combining all these different tools and you'll find which ones suit your need best. And I don't think it's like one model dominates them all. I think it's going to be different models for different use cases.

What are you expecting over the next year in terms of development with AI? How do you, what do you see coming that people should pay attention to over the next 18 months?

Great question, great question. Multimodal, I think that's something you touched on, you mentioned first for sure. That is going to be a complete game changer on what is capable from these models. And we're gonna see so many applications I can't even imagine, like we can't even like you. Like we could sit here for 100 years and we probably can't even come up with half the stuff that is gonna come out on Twitter in like, you know, six months from now.

Gotta put a dollar in the can. It's x.com now.

X.

No.

They got me again. Yeah. I think multimodal is going to be massively huge. Like once we get these models inputting images, inputting audio and for it to like work cohesively so it can like actually read the audio forms where it's not just like translating what the audio is and then feeding that into the context where it can be like, oh, I can understand the intent behind when you said hello versus hello, like those types of things. I think that's going to be like mind blowing. I'm so excited for that.

Yeah. Multimodals. It's so early, but it's so amazing to be able to. I was at the Carmel Mission and I took a picture of the mission and I sent it to OpenAI and it said, oh, you're standing in front of the Carmel mission. It's like, whoa. And then it started telling me all sorts of stuff about the mission.

Yeah, that's crazy. We're like living in sci fi. Like we're gonna have like little. Like we basically have Jarvis. Like, how cool is that? We have Iron Man 2007. Like we. Everyone has a Jarvis. That's awesome.

Development is not going to stop changing. I'm picking up tomorrow a chip designer who's building a new kind of neuromorphic chip, a new biological chip where he's using biological cells as part of the architecture of the chip, which is supposedly going to greatly increase the speed of that chip, particularly for AI loads. So that's coming in the next. Right. Cheaper processing. Right. Moore's Law is not dead yet, right?

It's. Yeah, it's crazy. Moore's Law has just shifted to how many tokens you can fit inside a context window. I think it's one of the things my friends and I talk about. Like we went over to the computer history museum down in. I think it's like it's in the South Bay somewhere. And we were looking at the chips and we were looking at the little eight like eight kilobit processors that was on this little intel processor. And we were like 8k, look how far we've come. And we're looking like, oh, like the context windows. Only 8K at the time was only 8K. We're like, wow, it's going to come so far in the next 20, 30 years. It's going to be insane.

It'll be interesting to see what happens as we get into robotics. Right. Where we're going to be talking high fiving our humanoid robot and things like, hey, can you take out the trash? Or can you, can you learn something new on our manufacturing line? At first, someday it'll be like, hey, can you stand here and watch me cook a meal? And in one time, it's going to learn how to make your grandma's favorite meal, right?

Yeah.

To me, that's just. It's all science fiction, what we're living.

In right now, it's all science.

Where this all started, right?

Yeah, it really. This is like the most science fictiony past couple years, I think, like, ever. Especially with COVID where it's like, oh, yeah, pandemic. Yeah.

If you told me all my school teachers just four years ago, if you told me, oh, all your school teachers are going to teach your kids on Zoom, I would be like, like, you're insane. How is that gonna happen? I did know how it was gonna happen with a pandemic, but. Okay, here we are, right?

We're in a world, right? New world, new world. It's amazing.

It's gonna be fun to watch you and talk to you over the years because you're like building the shovels for the gold miners, right? You're helping the people who are gonna build the robots and build the autonomous cars and build the new financial systems that are using.

Yeah, that's our plan. We're going to be there. We're going to grow with our customers as well. Sorry to cut you off, but yeah, that's our plan. That's cool.

Thank you so much. Is there anything I should be asking that I'm not asking? Because I'm not a developer, I'm not down in the dirt here, and I talked to a lot of people who are. Therefore I can ask some questions, but what kind of question did I miss?

Let me think about this for one second, because this is a. This is a good opportunity to share what's going on in the space. I don't know if I can answer your question directly, but I'll leave you with this. I think I was talking to some founders or some people doing stuff, and they were saying, oh, man, you got in early with a Helicon, which granted, I did. I would like. We had first mover advantage. It was really nice, but I don't like. I don't think now is not early. That was six months ago. That was eight months ago. Hasn't even been a year since we worked on a helicopter. Like, now it's still so early. Like, I would. I would highly recommend. We're at an inflection point in history where if you want to build something, it's never been easier to go build something because you have ChatGPT, you have all these amazing tools. You can build something that's never been made before. Like for the, like the amount of new capabilities has just increased like a hundred X. And if you're a builder or if you're not even just a builder, if you're people, if you're someone that make things like you're a psychiatrist, for example, do it. And it's not early. I think that's the main thing is it's. Sorry, it's still early, it's not late. It's still like the. We're on like day zero of all this stuff happening.

The, the real challenge for a lot of these companies. There's 4, 000 companies in AI that I've found on X. I assume I'm not finding them all, right? So there's another two that I don't even know about.

There's a few hundred in YC too.

And so this is a huge number of new companies. They're struggling to find customers. Right? Getting that product market fit, getting a business going. Right. That. It seems like that that's the, a huge challenge for this new community of developer tools makers, right?

It's a huge challenge for sure. I think the main. It was like a meme with all the developers in San Francisco, which is what's your defensibility? What's your defensibility? Because no one had it was we're all chat GBT rappers. Like, we have no defensibility chip. OpenAI could just release plugins and destroy half the startups, which it did. And I think the. Yeah, I think a lot of the other problems with people that are building apps is they usually find a solution and then build it first and then be like, oh, who wants to use this? Which is. It's so classic, but it's so obvious. Find the problem first and then build a solution. Don't build a solution first. Like find. And what I mean by find, the problem is find somebody who is willing to use your product and then build around that person. So for us it was, it was all of our friends that were building LLM apps and we were like, we asked them, we were like, hey, we're going to build this monitoring tool for you. Can you use it in your company? And they were like, yes. And we did that for five of our friends. And then we were like, sick. And then we did. And that's how we built that helicopter. I know that was a tangent, but I have a lot of passion toward that feeling.

Where do we learn more about you guys?

Helicon AI H E L I C O N E AI Also follow us on X. Yeah, thank you, Robert.

Giving me a development.

It was really fun. Yeah, you're a lot of fun. I appreciate you, Robert.