Training the AIs' Eyes: How Roboflow is Making the Real World Programmable, with CEO Joseph Nelson

Hello, and welcome back to The Cognitive Revolution. Today, my guest is Joseph Nelson, CEO of RoboFlow, a computer vision platform that supports more than 1 million engineers and more than half of the Fortune 100 as they seek to turn proprietary image and video data into a competitive advantage. We begin with an overview of computer vision capabilities today. Joseph notes that while language is fundamentally a human construct and inherently optimized to be understood, The real world contains a fat tail of chaotic scenes which are not at all optimized for understanding. And thus, just as the Vision Transformer came about 3 years after the original Transformer, computer vision today is roughly where language capabilities were 3 years ago with the introduction of ChatGPT and GPT-4. Which is to say that while frontier models can do amazing things, and most problems can be solved if you're willing to put in the work to fine-tune and pay any inference cost, We have a long ways to go before foundation models will really be able to do it all. To make this concrete, RoboFlow maintains a site called visioncheckup.com, which highlights the spatial reasoning, precision measurement, and grounding failures that still plague even the best multimodal models today. And importantly, even when frontier models can solve a particular task, you can't wait 40 seconds for a reply when you're powering instant replay at Wimbledon or monitoring for defects on a high-throughput manufacturing line. And so there's often still a lot of work left to do to get vision models running efficiently enough to meet production latency and edge deployment requirements. This is where RoboFlow comes in, and I was super interested to hear Joseph describe what it looks like to go from an open-source vision model to deploying your own task-specific model today. He emphasizes first the importance of establishing clear requirements upfront because the performance thresholds the different customers need to hit on their respective use cases can vary really quite dramatically. From there, the process often involves distilling frontier model capabilities into much smaller models, like RoboFlow's own RF-DETR model, which they derived from Meta's Dino V2 backbone using a really interesting training technique called neural architecture search, which in turn uses a weight-sharing technique to train thousands of network configurations at once, all within a single training run. This process ultimately produces a set of models of varying sizes that collectively map out a performance Pareto frontier. And today, RoboFlow has productized this approach so that anyone can now run it on their own dataset and come out the other end with an N-of-1 model that is optimized specifically for their problem. From there, we cover a number of additional topics as well. Joseph explains that Chinese companies have consistently led in computer vision and how much the American open-source ecosystem currently depends on Meta, but also why he's optimistic that NVIDIA will fill the gap if Meta's new AI leadership changes priorities. He also describes how coding agents are expanding the market for RoboFlow's tools, how skills are emerging as a new go-to-market vector, and how RoboFlow plans to use a first-party agent to guide users through the process of building computer vision pipelines. We also discuss the state of AI's aesthetic taste and why the inherent subjectivity of aesthetic preferences makes this such a hard problem. We hear about the emerging S-curves that Joseph is watching, including world models, vision-language-action models being developed for robotics, inference time scaling for vision, and wearables, which are now selling millions of units per year. We get his vision for how computer vision contributes to a good life as AI matures, which includes everything from precision agriculture and food safety to self-driving commutes and real-time sports analytics. And finally, he explains why he worries that overly opinionated regulation could accidentally stifle all sorts of surprising but valuable use cases. And why he recommends that policymakers focus on outcomes instead of trying to regulate the tools that people are using. When it comes to computer vision, Joseph has quite literally seen it all. So whether you're looking to catch up on the field like I was, or looking for a practical framework with which to approach a specific challenge, I think you will find a lot of value in, and I hope you enjoy, my conversation with Joseph Nelson, CEO Joseph Nelson, CEO at RoboFlow.

Welcome to The Cognitive Revolution. I'm excited for this. So regular listeners will know I really got into AI in a full-time obsessive way in my role as founder of Waymark. And it was such an exciting time 4 years ago when things were just starting to work. I ended up going really deep on what was available in computer vision at the time with Clip and Blip and Blip 2 and Clip Embeddings and trying to figure out how— the problem that we had at the time was we have all these small business users. We had developed a pretty good technique for scraping their websites and their kind of online presence and creating an image library for them. But then what to do with that image library, right? It was just initially just a total jumble of photos. We couldn't make any sense of it. We made very blind guesses as to what we would actually put into content for them. And that obviously had a, you know, long way to go before it really started to work. So I had a ton of fun in like 2022 into 2023 timeframe, getting deep into the weeds on that stuff. And obviously a lot has happened since, so I'm really excited to catch up on a few years of computer vision progress in 90 minutes or so. Maybe let's start by just kind of setting the stage, like where are we today in terms of computer vision? You can come at that from a lot of different angles.

Yeah.

Maybe start with like use cases. You know, what are the use cases out there that are really well established, that are driving the most volume, that are driving the most value? Give us kind of a survey lay of the land.

Since you brought up CLIP, maybe we can start in terms of some of the research that's powering what's now possible, and then the sorts of use cases that flow from there. With vision, it's funny because when you think about AI and the trends of machine learning, originally a lot of it, vision was home. You had ImageNet. You had MNIST, and deep learning gave rise to, is that photo a cat or a dog on the internet? And then you had language, I would say, almost jump out and take the lead in terms of wow factor and in terms of understanding with the introduction of the Transformers You Only Need paper in 2017. And then you almost have 5 years of language-like cooking with scaling laws and Chinchilla and GPT-2. And then you start to get language products in GPT, I would say 3 and 4 is really where things started to break out, and ChatGPT in 2022. That 5-year time delay of introduction transformer to products that become used by nearly a billion users, I think every single week now, is now happening in vision because you had the vision transformer get introduced in 2020. And so that ends up being another stepwise change of what's possible and what capabilities are easy or easier maybe out of the box. But to your point, what's interesting is historically there's been this divergence of is this a language problem or this is a vision problem? And modalities are crashing together because just like our brains, you get more context if you can use language and vision together. However, there's some pretty meaningful differences of visual understanding, both in the way that visual models work, but also the use cases of where it's most impactful. And I'd say actually one of the biggest ones is I think about the analogy of the way our brains work as a useful way to inspire how our systems for visual reasoning will work. As in, we have this big LLM reasoning engine in our heads that is our brain, but we also have this rods and cones and visual cortex that operates and makes decisions in— we jokingly call this like your lizard brain, like fast reaction, way of understanding the world. And biology has evolved to having specialized systems for visual understanding distinct from broad-scale reasoning, and the amount of neurons dedicated to that being more than any other sense. And I think the same will be true from biological inspiration for the systems that get used in machine learning. So what does that mean in practice and that abstract idea? It means a lot of stuff runs at the edge. A lot of stuff runs low latency. A lot of stuff runs out in the real world. So for example, like in a lot of language or multimodal or multi-agent reasoning problems, you can have the benefit of assuming you have maybe near infinite compute 'cause you can run a long-running job on a data center. A lot of visual tasks where vision is most useful tends to be where you don't already have a human or eyes on the problem. You're understanding an environment, maybe in a remote location, maybe a manufacturing line. Maybe it's you're shipping a product. Maybe it's you've got cells underneath a microscope. Maybe you're looking through a telescope and discovering new galaxies. Maybe you're building robots. And for a lot of those use cases, not all, but many, you need fast reaction times in addition to large-scale reasoning. And so you see this increasing divergence and specialization of where vision is especially helpful for low-latency tasks and for things that are, it feels maybe intuitive, but out in the real world, like systems that we want to observe, where LLMs and language is inherently a human construct, but the visual world isn't inherently a human construct, right? Like, language only exists where people do, in systems that humans have crafted. The world's much bigger than just language. And anecdotally, the amount of distinct scenes in a day is more diverse than the number of unique words you probably read in a day. And so that heterogeneity, that richness, Makes visual reasoning, I think, harder. It means that the long tails are fatter, and it means that the use cases tend to be out in the world, for lack of a better way of describing it. So the use cases that we see, we become a natural sample of where visual AI and computer vision is being used in the real world. Like about a million devs download open source every 30 days, about half the Fortune 100 build on the platform. So we have this kind of insight of what's actually making its way to production. Where are people tinkering? And, you know, it tends to be these things that are, I'd call them operationally complex problems, maybe like in the enterprise sets of use cases. And for platform, you get broad amounts of inspiration, which could be like a hobbyist that wants to understand, like, I don't know, I like to play board games, the dice that you just threw. I swear every time I play Catan, my numbers to get drawn the most. So I want a camera to prove to my friends I'm the most resource efficient compared to the resources that I drew for the way the dice came out, a serious and a joking example. Or we've, there's this YouTuber that maintains a channel called Dave's Armory out of Canada, and he's built a flamethrowing weed-killing robot with RoboFlow. Or he's built, he built his son like a self-driving couch that like follows him around the yard, like these silly things. And then you have more serious things inside the cases where like powering instant replay at sports broadcasts at Wimbledon or doing quality assurance in products that are being produced at Rivian, or any sort of like physical world thing. But one thing that I like deeply believe, and I think the rest of the world's kind of coming around to this, is that visual AI, visual understanding, and at least even that part of multi-agent reasoning, is gonna be bigger and more important than just language. As in the ways for AI to reach its most potential, it needs to be out in the world. It needs to understand, see, reason, because the world's a pretty big place. The universe is even bigger. And for the systems that we want to use and rely on, they need that type of capability. So linking that back to the research that's progressed, there's a lot of work to be done, but there's been a lot of progress in, from CLIP to present that we can talk a bit maybe in, in detail on. But to set the stage, I would just basically be one of like really optimistic that we're approaching the ChatGPT moment for vision. And the infrastructure to power all of that is coming online, which means you're about to see a Cambrian explosion of all the places and consumer expectations are just gonna be disappointed absent the ability for folks to have visual understanding in the products and services we use day to day. So that's like maybe how I'd think about what's going on and what use cases underpin the types of ways the research is making its way to production so far.

When you said that you think we're coming up on the ChatGPT moment, I thought that was quite interesting because in, sometimes I give talks about to kind of a, an audience that I'm trying to catch up on what's going on with AI. And I often give them the MNIST example as like, you know, look how simple this is for us, but we still can't write explicit code to identify these handwritten digits. As simple as that problem is, there's, there's no explicit algorithm for it even today, right? Okay, that kind of gives people a sense of why it is that we need this sort of fuzzier kind of intelligence. And then I zoom forward to the GPT-4 system card, and I show the image that I'm sure you're familiar with of the guy hanging off the taxi in New York doing ironing on the back of the taxi, and show them how we went from an ImageNet breakthrough in 2012 to that capability 10 years later, where the model says it's unusual to see this guy hanging off the back of the taxi doing ironing. So I was going to ask, and it sounds like your answer is going to be no, but I want to get a lot deeper under the hood on that. I was going to ask, to what degree could we consider vision almost a solved problem? Are there things— and I know that not everything works immediately out of the box. Not everything is going to work maybe at the cost profile that you'd want or the latency requirement that you have. But if we started off with just like a, is there any vision problem that we couldn't solve if we really put our minds to it today. That's kind of, that was the working definition I had for solved problem. Do you think we are not there? And if we're not there, why aren't we there? What can't we do yet?

The way I think about something being like solved problem is I think it's like I just asked the model and it almost impresses me. It delights me that it already understands and can do the thing I asked it to. And so I think that's why ChatGPT was such an aha moment for folks because you no longer had to train a model to understand sentiment or describe text or whatever. Just like I talk with it, it talks back. It feels like talking to, I don't know, a first grader perhaps. And so in vision, solved problem, I think there's a subset of places where that's true, but it's not nearly as— solved as languages. And I think the reason for this, in my mental model for what it takes to get us there, is what we were mentioning earlier of the world's very heterogeneous compared to language. If you think about this in a very, like, first principles way of the amount of data that it takes to encode text with Unicode, like, I can represent all of text in Unicode in memory much more efficiently than even representing a single image because I have 3 color channels, 0 to 255, RGB, pixel by pixel. And so that data disparity of how much more information it takes to even encode a visual scene, I think is maybe like an anecdata example of why there's more heterogeneity in understanding visual scenes. But in a maybe a more thing, it's again, the number of scenes in a day is different than the number of words that you read in a day. So my mental model for this is I think about the world as your standard bell curve distribution. And in the fat center of that curve, what we're measuring is the frequency with which a thing exists out in the world. If you went out, let's say, and you took a walk, or maybe just throughout your full day, and you write down, we'll say, every object that you see, and then you went and looked back at your notes and how many objects you saw and maybe how long you looked at them, you would have something of a bell curve of the things that you saw that repeatedly showed up, person, car, food. And there would be some things that are like longer tail, if you will. I don't know, maybe one day you were changing your oil, so you were under the hood of your car. Now, even that's something that you're not gonna do every day. So vision, having a model that can reach into those long tails, because it's heterogeneous and because I think those tails are fatter, It's taking just a bit longer to have the data represented as much as the models that can reason about what all the various different scenes and videos exist out there. So what does that mean? Some things are like, quote unquote, solved problems, like count people in this image, or increasingly OCR feels like a solved problem. There's a model, GLM, recently that we're really excited About, of it can run real time. You can query it with, hey, how much was my salsa from this receipt or from this Google Street Images? What's the house address on the left? And it's able to visually reason and extract and pull the correct answer almost always. And so something like that feels closer to a solved problem. But the nature of how diverse some scenes are is It's gonna take representation and probably some reasoning models to be able to reach into those long tails. And what's happening, I used to say in slow motion, but it feels faster. What's happening in vision is you're getting models, multimodal models, and this is like the big LLMs like Gemini, just as much as open models like MOLMO, just as much as models like the DeTR family of transformers that are increasingly pushing outwards on this. visualized bell curve where more and more of those things that you see on a given day are understood zero or multi-shot. And so it becomes maybe a semantic question of what you consider a solved problem. Like if you're in the middle of that bell curve, yeah, it's a solved problem. But if it's like something where it's so impressively surprising and delightful, it becomes a question of how long until someone starts to query and ask for things that— wouldn't have been represented and trained. And so I think that we're on that, we're riding that curve, like the expectation is getting faster. Now, one other complexity in vision is what we talked about earlier, which is a lot of vision is edge-constrained environments. You want answers now. You're running a webcam, it's on your phone, it's in the palm of your hand. And so that means you also don't have the benefit of maybe waiting 40 seconds for a reply from a model for the thing that you were interested in querying, which also means, it doesn't mean that the problems are intractable, but heuristically, I see maybe an 18-month delay between a SOTA capability from multimodal cloud-available model to something that you can get to run on an edge device, which maybe here we could define as maybe like a Jetson Orin level of compute, or maybe even like an iPhone where it's opaque, the exact GPU comparable you would make. So those things make vision feel unsolved, but I still think that what's gonna continue to happen is the expanding nature of that, that bell curve. And if you think about that mental model for where we're going with visual capabilities, then I think that's a good way to think about where the field's headed and frankly, the problems to solve.

Hey, we'll continue our interview in a moment after a word from our sponsors. So I wanna work through that kind of Pareto frontier of performance and cost latency, you know, and where it can run trade-offs. But let's do one more double-click on kind of the, the most expensive end of that curve, which is the cloud-available frontier models. I think you described them as, obviously these things are like famously spiky.

Yeah.

I would say it's been a while for me since I've had an image use case where I was like, oh, this thing can't do it or can't see it. I do remember some of those not too long ago, for example, with the Arc AGI puzzles. I remember trying frontier models on some of those puzzles, getting strange results, and then kind of working my way back to, can you just describe what the starting state is? And then I was kind of like, oh, well, no wonder it can't do the problems. It can't see the starting state accurately. It can't accurately just define which boxes are coloured what colours. But that's been a while, I guess. I don't know. I'm sure you would know if we just take our AGI puzzles and put them in today, are they accurately perceived? And are there other things that would be good kind of representative examples of spikiness where people might be surprised that, oh, I kind of wouldn't have guessed that Frontier LLM wouldn't be able to see this the right way? Also, do they work with few-shot prompting? Obviously, few-shot has been a huge unlock in general, but does it work for vision? I really don't know that. So I guess to sum that up, can we go one level deeper in terms of the capability profile of the friendlies? Totally.

Yeah, we spent a bunch of time thinking about this and then trying to help people navigate what their expectations should be for the problem that they're solving and where they may be able to have a zero-shot, multi-shot problem, or where you might be in a world where you need more representation of that problem before you can count on your model. The places where there's most frequently gaps. So one thing that we maintain is you'll see, just as you described the Ironboard example, you'll see these vibes-based evals on Twitter. We started rounding those up and throwing them on something called visioncheckup.com, which is if you took your LLM to the optometrist, what would it do and not do? And so we continuously update that thing. You'll see things consistently ticking up on that. But they're not all at 100% in the types of tasks that people like to try, including ourselves. So what's the common types of failure and where are you most likely to be disappointed? One of 'em is in grounding in particular. Grounding referring to like segmentation and detection, traditional tasks. But if you wanna say in your example, finding the starting position in ArcAGI, or sometimes I'll try to do like crosswords and I'll be surprisingly disappointed of the model's ability to know where the word goes in the crossword. However, if I just treat it as just a text problem and I say, hey, here's the clue and here's some of the letters that I know are in the word, the model will almost do better if it just thinks about it like text, if it doesn't also have to think about where in the puzzle that issue is. Or measurement, basically you can think about this subset of problems of things that are inherently very precise, where there's lots of precision, involved. And some of this is, I think, the result of the post-training that's applied to these problems. And a lot of the labs— it's a little bit hearsay, but seems to be increasingly common knowledge— are not as interested in just solving the segmentation problem. They're interested in solving what was the user's intent, and if segmentation is a tool call as a part of that intent. But even still, the segmentation portion of that chain of thought is pretty unsolved because there's so many different things that you would want to measure, see, and have pixel-perfect representation of. You will see that when you take more time to reason, AKA you do more tool calling and you find more maybe specialized expert models for the scene you're looking at, you'll get better results. But in general, I would frame grounding as still a pretty difficult issue with precise pixel-level The second place where I think there's disappointment on the Pareto that you describe of accuracy speed is actually speed still. Like the, the, I was using Gemini 3 the other day to try to like automatically label a bunch of data for me and prompting it, and it would do, but it would take 40 seconds each time. And interestingly, the non-deterministic nature of generative AI also led to some pretty difficult downstream results. Because for that example, like, I wanted really precise, consistent, not necessarily what you think could be correct or your best guess, or even not consistent results. And so that's another challenge is that like you and I could go try the same problem and get different results from the same model at the same time of day. We maintain this other property called Playground, just playground@rofl.com, where you can do SAM-3 versus Gemini versus Claude Opus. And what's really funny to me is that I'll find these failure cases and kind of report them to our team and then they don't reproduce. And it's actually not because of our use of the models, the model itself doesn't reproduce the same way. So that continues to be a little bit of a challenge after speed, I would say, is the reproducibility challenge. And then, I mean, it's a little bit redundant, but we talked about just the representation. You get into those long tails, And this becomes a function of the type of question you're asking, but there is still a lot of the world that's just not understood by models. And a close cousin to these problems is spatial reasoning, of being able to articulate not just the pixelized segmentation, but where one thing is with respect to another thing. So those are common sorts of failure patterns we see. Now, you had a good follow-up, which was, how does few-shot help address these things versus zero-shot? And the answer is pretty good, but still not infallible. Like, it helps, and how much does it help obviously is an interesting question. So one way that we think about these problems is we introduced a benchmark at NeurIPS this last year called RF100VL, RoboFlow 100 Vision Language. Basically, so folks that use RoboFlow for research will share their work in effort to build upon others' work and bring the whole community of computer vision forward. So there's a large set of open source datasets that folks can learn from and try and accelerate their own problem they're solving. We went and worked with users and researchers and the hundreds of thousands of folks that are sharing open source projects to move the whole community forward. We created a basket of 100 of them of like problems that seem to be represented in visual AI. And the domains that broke down into were problems like industrial, healthcare, flora, fauna, documents. There's a miscellaneous category, 'cause of course it's tough to put everything into a single bucket. And we evaluated Gemini and Sam-3 and a number of, uh, and OpenAI, of course, a number of multimodal LLMs. And then also models like OwlVit, which is a model that supports few-shot prompting, an open-source model. Alvit 2 is the most current one, and a model called Grounding Dino, which is the new version of Grounding Dino behind an API, but still more open in general. And basically the net here is we evaluated them by saying, can you do successful segmentation the same way if I passed a human annotator these same instructions? How does the model do? How would the person do? Of finding all blank things in an image based on those domains. And the best model at the time we published the work, it was Gemini 2, but got 12.5%, like, of 100% across all domains. And so the gap of how far these models have to go on, and this data wasn't, again, it wasn't like arbitrary data. What's really interesting is that this sample wasn't like a perfectly curated research dataset, imperfectly, like CoCoA or Object365 or these works that are very helpful contributions to the field. It was, these are the places that folks are using models. Now, a second thing we did to— so the zero-shot performance, 12%.

Cool.

Interesting. The second thing we did is we ran a competition at CVPR on 20 dataset subset, just so that for compute constraints, because we thought we could get the point across. So RF20 instead of RF100. And we said, if you had few-shot, that is, 1, 2, 3, 4, and 5 image examples, how much do you see the models improve and progress by comparison to one another? And the lift there, I think maximally was around 10% for a single model. I'd have to check the average across all domains, which is meaningful, especially when you're starting at 12%, but not a panacea, right? It's okay, great, like I'm helping ground the model, with the domain that I'm looking at, but it doesn't solve all the problems. I will say that's a place that I'm bullish. Specifically, I'm bullish about few-shot for visual problems and providing prompts perhaps as image-text pairs or even just as images with the task you're interested in, whether that's grounding or description or measurement or what have you. And so the story is clear, which is We need continuous, better representation of the real-world problems people are trying to do. And we still have a bit of a ways to go as a community for it to be totally solved, but progress is happening pretty fast. So that's the progression of maybe where things are.

Hey, we'll continue our interview in a moment after a word from our sponsors. Okay, well, that's a good start at the top of the curve. Let's maybe work our way down the curve. I mean, there's obviously multiple reasons that one wants to go down the curve. You may even add to my list, but obviously faster response time is a huge one. Lower cost is another great one. Ability to run on the edge is another great one. Ability to not have to send your data over the wire is another great one. There may be even more beyond that that you would highlight, but take me from kind of I can naively send images into one of a few frontier APIs, maybe with few-shot to like— and maybe you could take— I'm not sure which way makes more sense to organise it. You could kind of go from large models to smallest, most able to run on edge, or maybe a different way. Maybe they line up, but maybe there's a difference in how you would think about actually coaching people From starting with, well, why don't you just— at least what I normally do when I have a new challenge is, well, let's just see what a frontier model can do out of the box. Now, once I've calibrated myself there, I can approach optimisation and fine-tuning, whatever, in any number of ways. So I'd also be interested in if it's not the same as the kind of the curve of trading off convenience for all these other goods of latency and and cost. I'd also be interested to hear how you recommend people navigate the path from that kind of first naive baseline performance to where should they go next? What model should they go try to open source? How much— try to fine-tune? How much data are they going to need? What techniques should they use? If that doesn't work, then what do they do until they're finally in some happy place where they've got everything that they wanted?

I appreciate the way you broke that down. You're like, There's the speed accuracy curve, which folks know those things trade off. A bigger model takes more compute and is slower. And then there's these other dimensions that don't fit neatly on a graph, but might be really important to somebody. Like you wanna maybe own your AI, maybe you wanna be building your own IP as a business. Maybe it's important to you as an individual. Maybe there's a constraint of the business case itself, of the problem that you're solving, where again, low latency is super, super critical. That could be also a privacy consideration where, as you said, you wanna keep the data. Local to your thing. There could also be security as a close cousin to that. So there's a number of things that kind of frame where the, like where someone is gonna fall along those axes. Now, how to navigate it and what I think often matters is it stems from the job to be done, if you will. So making that be like less generic and more real and something that people can actually think about. What I think about is many problems, especially in vision, require there to be an instantaneous response, like a very— you have something going down a line, you're watching a live sports broadcast, you need a decision right there, right then. And so of course that puts someone into a real-time category. And so usually there's the need to co-locate compute or run something on the edge. And you already know that you're in a class of models where you want something that inherently, if you're gonna run it on the edge, you need to own it. You need to own the model, you need to have the weights, you need to put it into your environment. And that's where open source and where we're incredibly, we invest a ton in open source for this reason, in terms of publishing our own models, in terms of supporting open source repositories. I'm very optimistic about the future of open source AI, both Because I think it's an important way for everyone to realize the benefits from it, just as much as I think it helps discover bottoms up all the ways this technology is going to be impactful. Now, a pattern that we see that's interesting of navigating these two is similar to what you described is, is this problem tractable? Is this doable at all with the types of models and then maybe reasoning and nudges I would use for pre and post-processing of the model? And then fitting it to where I want to run it. And so we see this rising trend of distillation. So for example, SAM-3 is a promptable model where I can say, find all the sheep, find all the people, find all the hockey players in this image, and it'll do a pretty good job. It's not infallible. There might be perspectives you didn't include, but it's state-of-the-art for open vocabulary segmentation at a minimum. Let's pretend that you are one, one of our customers does the instant replay for clipping at Wimbledon in the US Open, and they actually bring compute to the courts because you have a live broadcast, you have sub-10 milliseconds to put something over the wire. And there are compute, there is compute that could run SAM 3, but in this case it just wasn't economic to get the compute logged of that size. So, Basically, you're in the situation where you want a model that you can own and run at the edge live over the wire to, in their case, frame the instant replay that you want to put on broadcast networks. Maybe you see on ESPN+ or CBS or something. And the things they wanna know isn't an open vocabulary list, right? So you don't need a model that can see, I could prompt Sam-3 for poker chips, I could prompt it for deer. The odds that those things show up at Wimbledon is pretty limited, hopefully. And the odds that I need them for my replay model are even more limited. So what I could do is use prior video from a prior year, prompt Sam 3 and say, hey, give me all the people, gimme the tennis ball, gimme the court, gimme the net. Have that then go and auto-label a dataset. Then I have a really high-quality curated specific dataset and I can train my own smaller model that I can run on the edge, a model like RF-DETR, which is the current state of the art for doing real-time segmentation and object detection. And not only does it run on the edge, but it's so efficient that I can run multiple streams on a single A100 in this case. And so I get the cost advantage. And then of course, if something's cheaper, you open up more possibilities. And so that's a common pattern we'll see is, is this thing, can I use a model to failure? And if it does work great, then can I make it be mine? Or can I fit it into a use case where I know I'm gonna need to run on the edge? Now, the other thing is that even as models eat more of the overall task, it's still okay, of course, to put a model in a harness or do pre and post-processing of the model to nudge it in the direction of what you would expect. There's nothing, no shame in like still using traditional techniques for post-processing. For example, there's models where I can just ask, like I could ask Gemini to say, count the tennis players on the court, and it would give me just the response of the count. But I couldn't just ask a detection transformer count. I could say, gimme the persons, and then it responds and there's 2 people found. And then I would add a tiny bit of logic. That's super fast code that just counts the class outputs, right? And so there's no shame in continuing to use stitched-together post-processing logic for the purposes optimization of speed or where something's going to, be possible. It reminds me of the database roaring. The most recent one was vector databases, but even before that, when you had a lot of SQL databases and then NoSQL databases, and where is it most useful to have document stores where you have unstructured data that references one another? Where is it most useful to have structured data or structured tables? And at some point in time, of course, you're gonna have to deal with sharding your databases if you have everything in those records versus if you maybe had a NoSQL database that's gonna scale for you automatically. And there's trade-offs in both those worlds. Reminds me of that where it's not a question of purely capability, it's a question of constraint of the job to be done at hand. And again, running things on the edge in real time or a model you own, or there's a, we didn't talk about cost, but like, man, streaming to the cloud for video nonstop can be expensive if you have quite a few streams versus maybe owned compute. So these are all things that drive reasons why you can use maybe max ceiling intelligence and then apply it to a system that becomes one that you own and use. And you've seen this trend in language encoding models too, right, of specialization and small models and expert models. And so in some ways, that's a place I think that language has drawn inspiration from vision, where there used to be consensus, it felt like, that it's one model to rule them all. Now it's increasingly maybe flipped back to actually there's domain-specific models and optimizations to be made. And I think that vision is increasingly in the camp of you do want a domain expertise model because you might be compute constrained of where you're gonna run your system. So I don't know if that's the color you're thinking about, like navigating the Pareto and considerations, but those are things that like we at least commonly see when we see folks approach problems like this.

Yeah, that's great. Could you maybe give a little sketch of the scaling laws, so to speak, By which I mean, this is a little dated at this point, but I used to have a talk where I kind of coached people through how to automate tasks, usually language tasks. And one of the big things was just like, you kind of have to think of order of magnitudes of data, right? So if it's not working out of the box, then go get me like 10 good labelled examples. And if that's— you could probably put that into context and then it might work. And if that doesn't work, then you might need to think about 100 examples, and you might have a small fine-tune on your hands. And if that doesn't work, maybe 1,000 will. But I wonder kind of what you see in terms of, I guess, first of all, what do people need in terms of reliability? And then what do they get out of the box? And then how many steps do they typically have to take up those order of magnitudes to actually get there? And maybe also, how does that relate to model size? Obviously, bigger models in general kind of can do more, but especially if you're doing very narrow stuff, it seems like you probably can get everything squished into pretty small models. So, yeah, what are the model sizes and how much data do you need to kind of step up to where people are hitting the thresholds they need to actually deploy?

The data question is one that, as you alluded to, gets informed by the business problem of how many nines do you need before you're able to use a thing in production. For example, if you're building a system where your alternative is you have no eyes on the thing, then you're probably more accepting of a less accurate model. Maybe, I don't know, you're trying to get a sense of attendance or you're staffing your retail location and absent vision, you have no idea how many folks are coming in day in, day out. Maybe you can go check your point of sale system as one source, But let's just— and that you have to try to get to some source of truth where some people are in the store at the same time, or not everyone maybe checks out, or there's a different way we want this information. Point is, in this case, we might not have any eyes on the problem. We might not know how many folks are gonna be present in our store at a given point in time, or maybe in our museum is an even better example, 'cause people don't explicitly check out. So it's, I just don't know. And so if you add a model that is counting and it's 80% accurate at counting, you might be like, great, ship it, put it in production. If we have a sense of, do we have a dozen people or 100 people at a given point in time, then I'm comfortable with that. Whereas— So that's totally valid. Whereas we have some other problems where we have like healthcare manufacturers that make critical life-saving products. You can think like the IV bags in hospitals. And for them, an escape defect, meaning like a piece of particulate matter making its way in a product like that is life-threatening, let alone the detrimental impacts it would have on the company's reputation and so forth. So with lives on the line, you need to be really high recall to find if there's any particulate matter. And you're probably comfortable with adding vision to augment whatever system you currently have, whether it's people or lab inspections or sampling methodology, and leaning into where you can get more reliable vision systems. So ultimately, like a lot of things, it's a business question for what level of accuracy tolerance you accept. Now, in terms of the number of images or videos that it takes to get there, this becomes a function of how varied the scene is of interest. So like on one end of the spectrum, you have self-driving cars that are in the big, wide open, crazy world. I remember Karpathy's talk like 5 years ago at CVPR, Where he was like, finding a stop sign, how hard can it be? How hard can it be? It's the same red octagon everywhere, right? And he's like, wrong. In photo after photo, he's— Here's a stop sign that's blocked by a bush in a parking lot. Here's a stop sign that says only stop if you're going right at the intersection. Here's a stop sign that's on a school bus. That's a stop sign that appears for a temporary amount of time. Here's a stop sign that's on a gate that when the gate is up, you can see the stop sign, but you don't need to stop. And you're like, man, something as simple and straightforward as a stop sign, there's tons and tons of edge cases to understand. And so that's navigating the world fully autonomously. And yet, of course, showing up and how long it's taken for us to claim victory laps on self-driving cars. And now that they're here, it's almost folks are surprisingly unexcited about it. Whereas you contrast that with something maybe like a manufacturing line where you know the thing you wanna make and there might be a finite number of ways that thing is made wrong. Like maybe we produce batteries for an electric vehicle company, and the defects aren't always the same defects. So it's like a tricky problem where traditional rules-based, like look at this image, do some OpenCV, if there's a deviation, then flag it. But, and so machine learning's gonna be helpful 'cause the defect could be like a different length or it could present itself differently. But the amount of variation that you're gonna see in the scan of a battery, the cross-section scan of a battery, is much more finite compared to driving on the open road. And so the orders of magnitude of data that you need, instead of talking about petabytes of video files, petabytes isn't even sufficient for a car. You can probably get away with hundreds, frankly, hundreds of images in the case of a controlled environment to be able to produce something of utility. The last part of what you mentioned is model size. And so, yeah, the intuition here holds. The smaller model, probably the faster it is, and also perhaps the less recall or precision it's going to get. Maybe the way to think about this is the RF-DETR family of models, which is the current SOTA for doing real-time detection and segmentation, come in nano, small, medium, large, XL, 2XL. And at the 2XL size, if you do a fine-tune, it is more accurate than if you fine-tune SAM-3 and 40x faster. Now, of course, if you're doing a fine-tune, you're inherently saying, I want this fixed class list. So it's a different problem task type, right? It's not open vocab. It's, I know the things I wanna see and I wanna know if those things are present or absent or how many of them there are. And then on the smaller sides of the spectrum, you can get like pico or nano models that are 180+ frames per second on a Jetson Nano with 4 gigabytes of RAM and of the current family. And again, based on the difficulty of the problem, if you're doing something simple like seeing oranges on a rack in a grocery store compared to finding particulate matter in an IV bag on a manufacturing line, you can get away probably with a Smaller model that still clears the floor of the business utility while being more compute efficient and delivering the results that you need. And actually, get away is probably the wrong framing. It's actually, that's actually an optimization. That actually might be your most optimal strategy 'cause you're able to deploy it at higher scale with maybe less compute. So that's like the, if you walk along the curve, it's— The good news is I think intuition holds here. It's not, it's, you would expect that Harder problem, more data, bigger model. Yeah, all those things follow what your expectations probably would be.

You mentioned distillation from kind of foundational open source models as a way that people are bootstrapping their way into datasets and then obviously fine tuning downstream of that. And you mentioned this RF debtor model, which the RF stands for RoboFlow, right? So maybe if I understand correctly, there's some I don't know if it's on that model in particular or in other places you've partnered with Meta. And I'm interested to hear the story. And I'm also interested to hear the lay of the land in terms of who is producing the open source models, why, and how. In language, of course, there's been a lot of talk lately about Chinese companies distilling from Claude, et cetera. and Anthropic trying to shut them down, which I think there's obvious business use case, business reasons for that. There's also questions around, like, what does that mean about the real strength that the Chinese companies have in terms of making their own models? Like, how much should we discount what they're able to produce? Does that explain why they're so spiky? And I kind of wonder to what degree this is also happening or understood to be happening. I'm not sure anybody really knows on the vision side, because I would say, aside from Meta, It sure seems like the public perception seems to be that the Chinese companies are leading in both vision tasks and also in, maybe not fully leading, but certainly in the open source category, leading in image and video generation. And so I'm wondering, is there some distillation going on there that they're kind of taking a shortcut, or do they have just tremendously better chops in that area? If Meta were to have a change in strategy and not be so interested, and obviously they have had some changes in leadership, Would the American side be kind of an empty bench? Are these projects so big that you, as RoboFlow, maybe could still dig deep and fund them on their own? Or do you need a hyperscaler partner like Meta to really get to the scale that you need? I guess geopolitically, strategically, and then you can dig into the partnership with Meta too. What should we expect in terms of Are we just going to continue to get these great open source models like manna from heaven, or is that maybe more precarious or kind of moment in time than people may appreciate?

I think open source AI, there are reasons to be concerned about its future to its past in terms of the number of open source models we're going to get. However, there's a lot of things that give optimism too. So in vision in particular, you bring up something that I think is underdiscussed, which is that in visual AI in particular, the US has almost never led, whereas in language we have consistently been ahead from closed models and open models alike. And there's a lot of reasons for that geopolitically, just as much as task emphasis and execution, but everything from the importance of manufacturing and vision in manufacturing and the importance of manufacturing in the Chinese economy. These are all trends that tell you why focusing on visual understanding and that as a domain is probably a high priority. But yeah, to name names of the folks that I think are in that mix, it's the Alibaba Quen team have done phenomenal work. They recently had in QuenVL and the initial Quen models, Quen3VL is world-class and competitive with even closed models in its vision language reasoning and scene understanding capability. The QuEN team has also had some recent leadership changes, or at least leading researcher changes, so that might be tenuous. The GLM team, we were talking earlier about their mixture of experts model, especially for OCR-specific tasks, but even just surpassing what's possible with closed models with their 9 billion parameter model, which is impressive work. And then there's the DeepSeek team, which they, if you remember, they published an OCR paper where actually the innovation was actually a data processing technique for LLMs where they gave a model a screenshot of a page as a way to get more tokens versus just each individual word in the form of readable text. And the realization was that the depreciation of understanding was much less than the compression that was experienced. So it was basically a way to give more tokens to scale up training. In the US, we're not without folks that are doing incredible open source work. You mentioned Meta, who are the publishers of the Segment Anything family of models. And I'd say they're the— SAM-3 is the best open vocabulary model globally, and Meta are the publishers of it. The Meta team, all the way back to FAIR and Yann LeCun starting efforts with Detectron-2, faster RCNN, introducing Detr, the Dino family of models. One thing that people dunk on Meta about is their lack of language models. And again, under credit how good Meta has consistently been at visual AI in particular and advancing computer vision consistently. And if you think about their business, this makes sense as well, of course, of making sense of photos and images that people share on social just as much as the future of glasses and so forth. You also have Microsoft with the Phi family of models that are multimodal. The Allen Institute with Malmo, though you want to talk about funding and turbulence, that's a topical thing that's taken place in the last little bit. On the diffusion side, you have Mistral doing some work on Black Forest Labs out of Europe. And then I think the other one that is pretty exciting is NVIDIA. NVIDIA has put a ton of effort investment into open source AI. I think they have the most open-source model repositories now just by account, if that's your rough heuristic. But the Neutron family models, Cosmos Reasoning, I was talking with one of their directors of open source, Nader, recently about how much they're investing in making those models be increasingly multimodal. And the Cosmos Reasoning team doing great work to advance beyond just a visual reasoning capability. So there is this geopolitical race for sure of wanting to have the best models and best possible. Now, you asked a question that is near and dear to me. Okay, so where does RoboFlow fit into this? As you're not a foundation model company, you don't have a data center the size of Manhattan like Meta, you're not the publisher of GPUs like NVIDIA. So where do you all fit in this mix? This is actually something that gives me an intense amount of pride, actually. So we published RF-DETR, and RF-DETR retook state-of-the-art for the US in a very specific area of important tasks. Real-time object detection and real-time instance segmentation. Before that, you had models like LW-DetR and the DiEM family of models, which are tougher to fine-tune, both of which are great work out of labs in China published. But RF-DetR, to tell you the story and how it ties these threads together, RF-DetR is the first real-time instance segmentation transformer as well as the fastest and most accurate for doing pixel-wise segmentation and detection. The way that we made a bet that initially I wasn't sure would work, and I've been delighted at how well it has worked, is we picked a narrow task and a small model that is useful on edge tasks, which just as you and I have been discussing, I think is comparatively under-addressed. And so basically we have this novel area where We know people need things on the edge. We know people want models to be theirs. We know that open source AI is under attack, and we know that's incredibly important to give people models that can run in environments that they might not otherwise have. So our the way that we did that is it marries these themes. We took a Dyno V2 backbone, so a pre-training from the Meta family of models, and they've since released Dyno V3, but Dyno V2 backbone, and we noticed that. There had been improvements from models in the Transformers family of passing accuracy but not speed for detection-type tasks. Similarly, there were some models on the end Transformers that were faster but not more accurate. And so we said, if we use a DynoV2 backbone and we use all the benefits of pre-training and we use a shared weights neural architecture search, NAS, strategy, can we select, intelligently search, and find the most optimal speed-accuracy model from an Object365 pre-train that then works downstream on COCO and user-fine-tuned tasks, and attach a segmentation head and a detection head. And at the time when we started these experiments, it was soon after we'd raised our Series B. So in total, we've raised about $63 million across All rounds, just to give you a sense of the size of resources we have available at our disposal. So not nothing, but also not all being spent on just this problem, of course. And also pales in comparison perhaps to the billions to be able to be spent on foundation models. And through the training runs, we realized that this technique has promise. And so we invest further in it. And we introduced the first detection model last April, the segmentation model in the fall. And we continue to invest in making the developer experience really high quality there. And critically, something I'm super proud of, it's Apache 2, which means that even there's been YOLO-friendly models, which we support and folks can use, but those are actually now not commercially permissible without a commercial license, which we're able to offer, which is awesome as a company. But I think there's places where people just want to build models that maybe they don't necessarily have commercial ambitions. And so it's world-class at what it does. And maybe to give a sneak preview, we already know some ways that that it can extend to perhaps other task types and even have yet more accuracy. The LWDETTR team in China have responded but not beat back some of our work. And so it's this cool kind of global arms race where your tiny friends at RoboFlow are putting the US on the map in a pretty big way. Now, if Meta stopped publishing open source tomorrow, if Nvidia started publishing open source tomorrow, just as I described to you, all of open source would take a hit because A lot of improvements are taking the best of ideas and experimenting, running ablations, smashing them together, and smart minds, certainly smarter than me, thinking about how to solve these outstanding problems. So it's something that, like, I think tells the story of what's going on in open source vision and something we're proud of just as much as, like, the problems that are yet to be done.

Well, yeah, I'm interested to hear more. And I guess if I was going to do one Double-click there. We're obviously entering, according to many, and I'm among them, the era of recursive self-improvement broadly for AIs doing the AI research. It sounds like you kind of dabbled in that a bit with this architecture search. And I wonder if there was anything, as you look back on that experience, that you found surprising. Did it feel like a brute force grind, or was there anything Are there any stories to tell about eureka moments coming out of that architecture search that felt somehow qualitatively different from a brute force grind through architecture space?

One thing that I think is really exciting is I'll go deeper on this idea of weight sharing in neural architecture search. So a lot of the times you're doing very brute force, train a bunch of different models, compare the speed accuracy of those models, And you're almost doing a grid search of different parameters that could help, an informed grid search, right? You're not gonna do things that would be, you would think naive, but it is fairly naive guess and check of train this model, see its speed, evaluate it back and forth. And there's still, of course, a degree of that. But, and we published a paper, so the details here are open for anyone to be, to dive in as well. What we did with weight sharing in neural architecture search is rather than training a separate model for every accuracy-latency configuration, We use weight sharing in NAS to basically train thousands of subnetwork configurations in parallel with a single training run. And so at each training step, one subnetwork is sampled by randomly checking parameters like patch size, the number of decoders, the number of queries, the input resolution, the attention windowing. We use deformable attention in the model. And then inference time, you can actually sample any of those subnets. What that does is it doesn't just mean that we've introduced like maybe one model. We've actually introduced a framework by which we can repeatedly produce open source models as long as you can do NAS against the architecture. And a NAS training run isn't as efficient as a single training run, but it's also not 7,000 times more inefficient despite having the ability to compare all of the number of different configurations. That was a huge freaking unlock. For us to be able to use our compute budget efficiently to be able to release models like this. And so that's one huge part of the story. The other maybe notable unlock was rewriting deformable attention, which isn't supported in every inference engine. So we've had to rewrite some support for it or wait, for example, for TRT in the NVIDIA ecosystem to support it. It now does, but that was a useful realization And then I mentioned the Dyno V2 backbone, which now that Dyno V3 is out, you can imagine what experiments we're running. So yeah, the weight sharing in NAS is something that's massive. And then by the way, anyone can use NAS on your own dataset and almost create like a one-of-one model for your problem. Because the way that NAS works is it's going to train and create an output, a Pareto that you can then pick from of like, where do I wanna exist along the speed accuracy trade-off? And so you can be like, yeah, I like, I can be anywhere along that curve within my available compute budget. And you could just obviously just max accuracy or be lesser on, on speed. And so when we saw that NAS worked on Objects 365, we were interested if it worked on downstream tasks. And now we've actually rolled out the ability to run GPUs in the cloud that'll do hosted NAS on any given dataset. And to the theme of owning your own AI, Like if you NAS on your dataset, literally no other model architecture exists that mirrors your dataset. Like it's like a one of one, which is like, I don't know, there's, if I was smarter about crypto, there's some interesting crypto thing there of a one of one to give somebody, but that's outside my wheelhouse for sure. But it is the most maybe purest form of like your model, 'cause literally no other model would've landed at those optimizations for the dataset that someone wanted to train on. So NAS was like, kind of the unsung hero that is a huge unlock for the efficiency gains we were able to see.

Yeah, that's really cool. I think one thing I noticed you have done, I don't know if you've done it exactly for this yet, but when people hear about this whole NAS and the, you know, it sounds complicated, right? We're going to have a Pareto frontiers worth of models. I would imagine, channeling myself, I guess I'll just speak for myself, that sounds both awesome and complicated. But I noticed that for at least some things, you are following the trend that I see, I'm seeing everywhere these days of, here's a skill that you can just give to your Claude code and have it speedrun through the process of setting this up for you. So I'm interested in like, how easy is it these days to get started if I'm sitting on some esoteric problem and maybe some small amount of data? That I'm, and I'm like, oh, this, uh, RoboFlow guy sounds like he's got some pretty cool techniques. Like, what's the path of least resistance to go come out the other end of this tunnel, potentially not having done much work and have my own, you know, one-of-one model that's, you know, got its own Pareto frontier of possibility and all that good stuff.

For NAS specifically, I gave you the real stuff 'cause I think the audience would be able to want to dive deeper. And I'm a skeptical person myself, so I'm like, Give me the real, real of what's going on under the hood. And that's why I mentioned the papers out there. But as someone that builds products, we also have created the easy button where it's run NAS on my dataset and then boom, we spin up a bunch of subnets on GPUs, kick off the training job, show the results. And then what happens back for the user is here's your Pareto curve and it's one-click press where you want to be along that curve. That's for the human user. You mentioned the agent user, which is also an interesting place to spend some time. But the first thing I would note is that a core thesis of RoboFlow of why we build things the way we do and how we approach things is you want to be very interoperable and allow someone to progressively reveal complexity, but set good defaults. So you can almost think about the products that we build that wrap RF-DETR or wrap our inference server or use NAS. Um, by all means, someone could set up their own infrastructure to do training and reimplement NAS. And it's all out there. It's open. The thesis in that is that by making it be easier and simpler, you actually inspire and engender trust, like that the benchmarks can be reproduced, that folks know where things come from. And then actually ease of use as a guiding philosophy, that strong defaults can be set. So I can awesome have a model trained for me or a dataset that like helps get curated. Or on the inference side, which we haven't spent even a ton of time about, like we've made tons of investments into, if you're doing vision-specific inference, there's a lot of optimizations you can make and assumptions you can make to most efficiently use the GPU. For just the vision, for just the parts of the network that require it versus, for example, a resize where you can run that on CPU. And so there's tons of, and all that's open, like inference. If you pip install inference, it's an open source GitHub repository anyone can use. And also if we provide that as good defaults in the service, and if we're worth our salt at all, then we should make products that are easy to use. That's for the human user.

Mm-hmm.

Now you mentioned something that's really exciting in general that's certainly bigger than any one company, and that is agents becoming the biggest user. And what does that look like? And like a lot of companies, we're leaning into the idea that if you expose CLIs and maybe MCP, we haven't released like a Rowflow-specific MCP yet. Certainly we have lots of good CLIs and there's this own debate of is it MCP or CLI future? At a minimum, there's CLIs that all the actions that take place in the platform, Claude Code and Codex and so forth, can take on behalf of a user. So that you can say, go optimize a model for me and make that be easier for me. Now, something that we are investing in as well is you've seen a common trend of companies that build amazing infrastructure products. And here I'll give like a lot of credit to Vercel where they've done awesome job of making a common set of products for the frontend cloud and increasingly the backend cloud. And then they've layered on top of that now a v0 agent, right? Their product that will then you can chat with it and it'll build website and it'll choose good infrastructure for your problem. We've taken a lot of inspiration for that as a way to enable users similarly to chat with our Workflows AI agent where it's like, hey, I just want to say count people crossing the line or watch cars at the intersection or whatever it is. And the model, a lot of these problems, it's interesting, the hardest part is actually discern what the user actually wants, what their intent is. Once you kind of have a sense of what the intent is, then models can intelligently be like, okay, you wanted to count, like Nathan wanted to look at cars crossing the intersection. So is there a model that can already know cars? Probably SAM-3. Actually, cars is a COCO class, probably RF-DETR, and it'll be more efficient and faster and more compute efficient. Great, let's grab that. And a pre-trained model that knows cars. Okay, you said crossing the line. Can I ask the user? The intersection has multiple places you could have meant by cross. So where, what intersection did you mean? And increasingly you can be in this future where it's like if Joseph were sitting down with you side by side and helping you construct your problem and pick the model and follow the architecture, can we expose that as an agent in a democratized way, for lack of a better term, that anyone has access to folks who spend many hours a week thinking about these problems, being the guide and Sherpa? For building a given pipeline. So in the scale of things, it's like ease of use as a, with good defaults as a platform principle with revealed complexity progressively. Secondly is agents using CLIs to basically use the same sort of easy use stuff. And then third is that like a first-party agent, which we haven't released yet, but maybe by the time this comes out, folks will discover. to guide folks down that journey. So that's some of the ways that I like to think about building products that balance giving someone the satisfaction and awareness that it's built on good primitives while still being able to create products that are ease of use and allow folks to get to value quickly without needing to know everything about every subnet of a training run, for example.

A very particular question that comes to mind that you might help me with or maybe set my expectations on a little bit is, so my company Waymark, we make videos for small businesses, specifically really focused on TV quality advertising. So like your classic 30-second TV spot is like really our bread and butter. Customers have asked us from time to time, hey, could you help us with display ads too? Our customers kind of, we partner with a lot of like cable companies, media companies. So they are ultimately selling inventory, advertising inventory to the small business. We're helping enable that with the creative solution. They ask us about these display ads as well. And now we're getting to the point where we're like, well, maybe we could add that. You know, it's a, we can vibe code all kinds of stuff much faster than we used to, certainly. But one challenge that I used to have a lot, and I'm not sure what the state of it will be today as I'm kind of digging into this, is aesthetic evaluation. And I don't know if, Way back in the day, there was really just one open source model, or one open source dataset and a couple of open source models that were trained on it that seemed to do a halfway decent job of aesthetics. And by halfway decent, I mean I could tell which was the top and which was the bottom end of the distribution, but in the middle, it was very unclear which way I was headed a lot of times. And there was one company that had one too. I forget. Oh, gosh, what was the name of that? EveryPixel, I think it was maybe called, that had trained their own aesthetic model. These days we typically go to foundation models for that and we say, like, what's suitable? What would make the business proud? Like, how would you advise us basically on these available images to use? And they work pretty well. Definitely slow, definitely more than we'd like to spend in many cases to grind through a huge library of images that a small business might have. Is there anything in the kind of small open source world that would be able to tackle a problem like that? Or is that just still so esoteric that nobody's gotten around to building that foundation for me?

Aesthetics is a tough one for the reasons you described, where the types of problems that models can recursively improve against is you can benchmark it. And the second you can benchmark it, then you can scale a bunch of compute, and the bitter lesson takes hold. Aesthetics maybe is a little bit eye of the beholder of what's good, what's bad. Now, there are some places where, and so for here, I say, for example, even if you just take diffusion models, some people just like the way Midjourney looks more than they like the way ChatGPT looks, more than they like the way Gemini looks, or Nanominet, I should say, when it creates examples. And maybe the model you're talking about is like the LAION team had the aesthetics predictor model that helped evaluate some of these things because they also did some generative image stuff. So they also released their, I think, aesthetics evaluator stuff that wasn't out when I was first really struggling with this problem.

And it was that I think the timing of that was such that we had kind of moved already to foundation models, but that was definitely the best purpose-built thing I think I've still seen to this day.

One thing that I'm sure you're aware of that your audience might find useful as a way to reason about this is in the context of display ads, there are some services like Facebook, for example, where you're not allowed to have text be more than X% of the display ad. They find that it just reduces the quality of the ad for the end user and whatever the reasons are. And so you can have, for sure, of course, that's a great example of the distinction between does this ad feel good taste-wise versus rules-based? Is there too much percentage of this image that is text? The automation of taste and aesthetics and preference. If I could give you a thought of how I might approach that problem, it's a great RLHF problem where if you have a given client that you know their brand guidelines, you know their style, perhaps there's enough history of display ads they've run where you can get almost like a vibe check model that has been tuned for what they've done. And with foundation models, like you said, perhaps you can do a few-shot approach of, Hey, these are the ways that this person commonly likes to do things. Is this similar? Again, the big problem even with that approach is so much of marketing is being different. So if you're adhering to the brand guidelines, you might be stylistically following what you should have done, but you might be failing the top-order task, which is stand out from the noise. So short answer, I don't have a great zero-shot aesthetics model for you beyond, I think, the things you're probably already doing. Longer answer, I think it's a great example of the conversation you and I have been having about what distinguishes a task where you can train your way, post-train your way to victory with objective metrics versus ones where it's a little more loosey-goosey to benchmark and therefore lives outside the range of where toss compute at it, get better results.

Okay, so maybe in the time we have left, let's talk about Just frontiers, like what's coming next in any number of different directions. There's, of course, new architectures that people tend to get excited about, myself included. Things like Mamba and state-space models more generally. Once upon a time, there was like an explosion of vision use cases there. World models are obviously a big deal. I'm not really sure how to think about how they will relate to vision. We've got increasingly credible claims that people are going to start to scale up humanoid robots and put those into presumably factories first, but then businesses and homes not too far into the future either. What are the things that you are most excited about? What are the biggest questions that you have that you are kind of like, if this works, it's going to be a game changer, but I'm not sure if it's going to? We're all about scanning horizons here. What are the horizons you are scanning?

There's things that I think are continuation of trends that are working, and then some newer S-curves that we're starting to ride. Trends that we're continuing to ride are transformer everything. We talked about how the vision transformer was in 2020, and attention is all you need was 2017. So you've seen diffusion transformers, DITs, and vision transformers, VITs, continue to eat more and more tasks and achieve SOTA accuracy. RF-DETR is exactly that recipe applied to real-time. That trend is known and going to continue. Another trend that's maybe more nascent is self-supervision, especially in the Dino family of models. And Dino V3 kind of showed that you could have good latent understanding things as a backbone without having large amounts of supervised labeled data. And then you can use that image understanding downstream for tasks, whether that's detection or segmentation or captioning or whatever.

Can you tell what the unsupervised trick is there? I always tell people the big unlock for language was language itself is structured. And so if you just have a ton of language, like predicting what, given some text, what comes next, we've got lots for you to work with. Similarly with CLIP, right there, it turns out there was billions of captioned images. What is the unsupervised unlock for, for Dyno?

Okay, so the Dyno family of models, you have 1, 2, 3, 3, and they're all riding on this trend of self-supervision. The Dyno V3 model I think was trained on billion scale. I have to check the exact stat, but I remember seeing that it was similar to the number of images on Rubble Flow Universe and I was like, huh. There's something there. So billion-scale-plus images. And the observation is, if you start to have sufficient representation of given domains, then maybe intuitively, if you think about just like a human without being told what things are, you start to develop intuition for where and how structure should exist in a given scene. And that is understanding. Like, if you know that The lamp is on top of the side table, and often in a room with a bedroom. Then you have a set of reasoning, or excuse me, a set of understanding of a given scene, and you can use that understanding. Again, it's a backbone, so you can't just use— you can attach a classification head to DynaV3, you can attach a segmentation head to DynaV3, but alone it's just a backbone which has really rich latent understanding. of scenes. And so that's the unlock is if you think about just looking at a bunch of scenes, you're going to start to develop your own maybe pattern matching is a crude way to think about it. To be a bit more—

Does that by some sort of masking type thing though? Or how is it creating a prediction task for itself that nobody needed to label data for?

In training, and there's papers, so fortunately that we can like falsify and understand. They use self-supervision techniques, which is you typically take a student-teacher model, and you have a bigger model that's the teacher that validates the output of the student. And as you see the student continue to do well with predicting either patches or gram anchoring, then you continue to scale up the student-teacher recipe, training recipe, to larger amounts of data to understand more senses of scenes. So that was part of the training recipe. The way that understanding happens actually isn't that dissimilar from actually just the vision transformer itself, where you have— it's actually crazy that this works. Literally, these models, and there's different approaches, but they take patches of the image. And this sort of is— it felt very unintuitive. It still feels a little unintuitive to me. But if you have patches of an image, It's almost that you can understand the rest of the image even from individual patches, even if you treat those patches independently. So it reminds me, like, way back in the day, I used to do language stuff. It reminds me of, like, bag of words, where, like, you would have a document and you would count the number of times each word occurs in a document, and you can start to get a sense of what that document is about. It's the same thing that's happening with understanding patches of a given image. Now, there are other techniques which we'll use, and this is— that's also why, by the way, earlier we were talking about the struggles of spatial reasoning. That's why. So you'll have increasing techniques that'll use cross-attention and get a better understanding of where things are in a given image with respect to one another in the overall image. But that's like the core unlock is if you have a high number of images of various scenes, and you run verifiable, falsifiable tasks of fill in the blank or what would you also expect to be here or diffusion generation, and then you have a teacher that's able to scale or able to validate the student's work, great. Now you have the recipe for a self-supervised loop. You can plug in more data and scale up. And that's what they did for— they didn't release the dataset, but it was like a billion plus. I need to check that, but I think it was a billion plus images. were in the DynaV3 pre-training. So it's actually like really cool that, that, that works honestly. And, and that it's open and there's a good technical report for it. Okay.

Sorry to take you, um, down that rabbit hole. Let's pop back up to just more horizon scanning.

Yeah. Yeah.

Uh, JEPA type things that, you know, those are always, uh, hotly debated as to whether they're like the inspired future that few can understand or if they're kind of beside the point. I still don't know where I come down on that myself, but feel free to opine on that or any other. Really, I'm most interested in what horizons you think are the most important ones to be watching.

So we were talking about ones that we're already riding the known S-curve of transformer examples and of self-supervision and how patch embeddings work to create understanding. New S-curves that I'm excited that we're we, society collectively, are starting to ride. One is world models. Within that category, there's a various number of techniques, like you described, the VGEPA technique, there's the World Labs techniques. And the idea of a world model, and there's different labs that have different approaches, but the objective of a world model is, can we understand and reason about I'm trying not to use the word world in the definition, but like scenes and places and out in the world. It's a well-named thing. Out in the world existence with a new architecture. And if you think about that, it's okay, so what's new, what's different? You're inherently, there's different approaches, but you're inherently multimodal by default. You're thinking about some models will think about this as like a next scene prediction of video of, okay, right, I've entered that some approaches will predict the next scene. Some will think about it a bit more like diffusion of a single viewpoint. The so what for world models that, like, I'm interested in is, will world models give us true understanding with physics, with spatial reasoning, with open-ended tasks? that we can just start to use? And the answer is probably, but the more interesting answer is on what time horizon, and there I'm not sure. I think what's interesting right now is we're using world models, like I would argue maybe even Cosmos reasoning as an example of a world model. And you can use Cosmos reasoning for, at a minimum, it's boring, but like synthetic data generation. And at a maximum, perhaps you can use it to actually reason about and use something to navigate a given space. So world models is one category. I think about vision problems as read-write access. And I think about RoboFlow as a lot of things we think about, for what it's worth, is mostly read access. And then world models are a form of blending read-write access. Robotics is an example of write access to the real world. You are modifying and manipulating the real world with a robot. Of course, that requires understanding to then— you have to understand to— you have to have read access to have write access. In a scene. World models are exciting because they're an example of potentially blending that, and potentially you get the understanding zero-shot or multi-shot. You could even argue that some of Sora 2 was the underpinnings of a world model to understand what's taking place. But that's one trend. To give an overview of another trend, and certainly go deeper in some of these too, that we haven't said is VLAs, vision language action models, which is incredibly popular in robotics. And like a video language action task is you provide instruction and a robot typically is able to action on that instruction. Move my computer 3 centimeters to the left or something might be an example of an instruction you provide, and then a VLA would be able to action on that. And in that world, you have a number of emergent, like younger startups that are thinking about that. You have NVIDIA's Groove project, working on that. Google's is R2, where they're working on that set of VLA problems. And I think maybe another way to think about VLAs is like, it's a new task type, a new paradigm, and we should expect the similar things you and I were just discussing around different model sizes, different levels of generalizability. VLAs in some ways will need to be edge-ready, 'cause if you're gonna run it on an embedded device and have embedded intelligence, then you're going to need, of course, the thing to be at the edge and run real time. So I think VLA is like a trend that I think is emergent and exciting and perhaps still under-indexed on. And then the— this isn't unique to language or unique to vision, it happens in language too, but it's worth describing because it's providing visual understanding as inference time scaling and multimodal reasoning and reasoning in general. In a lot of ways, vision can be a tool call of a broader agentic system that wants to understand and describe how to do stuff in a scene. For me, I'll find a thing I'm using Gemini for all the time is a replacement for instruction manuals of what does this button on the remote do? The pilot light in my water heater went out recently and I'm like, okay, tell me about this specific model. That's somewhat a high-stakes task that One would wanna proceed with caution with. I grew up as the son of a farmer, so if I wouldn't be allowed to figure out how to do that, I would be probably excised from the will. Fortunately, me and my friend Gemini and ChatGPT, we were able to solve the problem. But that's a perfect example where I'm using visual reasoning in the real world, but interacting with it with language. But the post-training reasoning, there's probably a tool call there to do some search, figure out the water heater model, figure out the instructions you're gonna provide. And so all of that is in the category of you have a big compute budget and you can do post-training and inference time scaling to give better results. That's just gonna continue. That's just getting going. And you can start to think about, so you can start to think about that as giving rise to visual agents that you can set off to go do a task for you, organize my images for me, or figure out, like in your case, perhaps like there's one that's aesthetically able to do good categorization of things that are brand aligned and not brand aligned across categories of display ads that you want to do. And so that's going to give rise to— we can learn from coding agents there. When you can let something run unencumbered over a long duration with a model, then we'll get similar benefits from long-running vision agents that can understand scenes and do things for us with all the caveats of speed and latency included. So Those are some of the trends that are the hypey ones that I'm thinking about and paying attention to. Again, they all— what I try to do when I break these down is I break them down into my normal distribution bell curve of the world and figure out what's the impact, what's the implication, and where can people use them? Can they make it be their own? Where is it going to be most useful? Broadly, I think that recording time of this episode is well-timed because I think the vibes, the pendulum is swinging back. To vision because you hear the rise of physical AI, of multimodality, the rise of hardware, what's defensible in a world of SaaS always being written and code generation being simpler. And so that's putting more and more people thinking about the real world and hardware and ultimately cameras and getting things into those environments. And to me, welcome, water's warm, been here all along, infrastructure's hot, let it rip. And we're fortunate to be able to power a lot of that sort of stuff. But hype-wise, it's something that has me pretty excited about the amount of activity that's about to enter the space. So those are some trends and maybe themes to track that I'm looking after.

How about wearables as one other one? That seems to be— kind of brings a lot of these challenges together, right? Because if you're gonna have something on your face, it can't be too heavy, can't get too hot, but it has to understand what's going on around you well, or it's more annoying than it's valuable, right?

Totally. We started a partnership with Meta for the Segment Anything family models and now more general visual understanding such that, for example, when they launch SAM models, they're on Rebel Flow with day one support. And now we're helping them understand where the model can be improved and not improved and some work like that. For Christmas, my significant other got me the Oakley Metas. She was like, if you're doing this awesome meta work, you gotta be dogfooding their stuff. And it's my first pair of wearables that are mine. I've used ones for like Spectacles and I'm always like tinkering with stuff. And when the Apple Vision Pro came out, of course gave that a run. Am pleasantly surprised, like wearables are going to inflect. There were 8 million pairs sold last year. By point of comparison, 60 million AirPods. So Like, pretty good amount of volume moved. And the Oakley ones in particular are like targeted at like active activity. So I like to cycle, and you typically already have a pair of sunglasses on your face that are a bit bigger for cycling. I like to run, and they do bone conduction for music. They understand the scene. The AI on board is not there. You have to have your phone with you, and they're offloading presumably some amount of the heavy lifting to the phone. You can say, hey Meta, and then get some of the feedback. But again, like many things in AI, it's the famous expression, this is the worst it'll ever be. And now that it's useful enough to be in a form factor where this is a pair of glasses where I went on a ride Sunday with some friends, they didn't even know that they were glasses that had the ability to play music and capture media. And then they gave them a spin first time and were like, man, we need to get these for our next ride. And that like really gave me the sense that this technology's arrived. But yeah, the constraints of running on the edge, constraining the amount of power draw it's gonna have. What's funny is like, I, I'm a bit like Charlie running up to his football and swinging and missing on AR. RoboFlow actually started as like building AR apps, like before we even had a company in 2017, we made AR apps just for fun. And I was like, oh man, we've arrived. And like, how wrong was I on the timing of that? And then we came back at it in 2019 and we made more AR apps. And I think the big unlock is the form factor so that you don't have to have the glass brick in your hand. You can have a different thing. So I'm pretty excited about wearables. I think we actually— Snap also has their Spectacles. They were the first publicly traded company to mention RoboFlow in an earnings statement. So there was a special place for me when we did an integration with their Snap Spectacles for developers to create custom lenses and scenes you wanna understand. So if you wanna create like a custom Actually, we had someone count the number of stop signs on their walk. It was like a funny thing because they wanted to, like, I think, prove to their neighborhood that they were safe or something. And so there's something brewing there, but the big change certainly is that the hardware has gotten good enough and the consumer willingness to adopt is showing up in the numbers. I hope, I think this will happen. I hope that ecosystem stays open. or becomes more open, I should say, so that anyone can publish apps. I don't have any inside information here in what I share in that I would bet the strategy is that right now it's closed APIs because you want to curate the experience and have high-quality first user experience with the apps people can use. But I would bet that the strategy will be to open that up App Store-like or maybe even Android-like where anyone could sideload. And so I'm excited for that future, but— I think the hardware platform adoption precedes the software adoption, and we're just now starting the S-curve of the hardware adoption of wearables.

So if you had to zoom out, this is a big ask, but if you had to zoom out from all these various horizons that we've just been scanning and try to kind of tell a story of how vision impacts life in general in the next few years. And I do think it's hard to predict anything more than a few years out at this point. How do you think life changes? Are we all going around with always-on cameras? Is that normalised? Do we all have a sort of 24/7 retrospective video of our lives, maybe subject to some times when we choose to pause it? Are there other unexpected— changes to life that happen as these technologies get deployed that you think people are kind of sleeping on? I think AI is going to change everything to a first approximation, but I'd be interested to hear your take on sort of what particular contributions vision is going to play to that and how it will feel as we are actually living it.

Man, I would love to paint the optimistic future for you of what vision unlocks for us, like step by step through everyone's day. Like from the moment you wake up to having food that's been produced with higher quality, with fewer pesticides, because you didn't have to spray all parts of the field, you only just spray where you saw weeds, to, I don't know, maybe you had eggs for breakfast and you want those eggs to have been visually assured to be safe from Hens and all the supply chain to make its way there to your house, to your grocery store. Maybe you grab like your clothes out of the washer dryer, which for some reason you still have to say whites or colors, which is very obviously like a for-the-taking silly vision problem, to your fridge auto-stocking itself because it saw you were low on eggs in the first place. And so you didn't even have to go and it called the Instacart MCP. So you automatically had the food in the fridge. You take your self-driving car to work. There's zero accidents because all the cars are communicating. Communicating with one another, and it's faster than you've ever been because you're able to not worry about unpredictability of someone else's actions with network systems talking to each other at a car intersection. You have Wi-Fi along the way, so you're able to spend more time with your family because your workday already started on the way to the office itself. You're in the office and you're communicating with colleagues all across the globe, and you have perfect, pixel-perfect diffusion representations of them in the room next to you. Remote work, same work, same place. It's just all the same of what it feels like to collaborate, at least digitally. There's gonna be something to human connection still, but at least the representation from Zoom has taken leaps and bounds, orders of magnitude better understanding of meeting with other people. To the, I don't know, you get home that night and you watch Thursday Night Football and the stats are real time and your fantasy team wins because you have the best algorithms to know Who was gonna play and who was gonna score. And you had your vision agent running in the background to do that better and faster than your friends. You have a package that showed up on the right time, in fact, same day, because all the vision systems that were in the factory and inventory to make sure the product was made right, checked in at the right places. There was a bot that delivered it to your door so that it wasn't strewn about. And your Ring camera made sure that there was no porch thefts or whatever that took it from while you waited for it to arrive. All the way to the moment that you brush your teeth with a, it's not silly, but an AI-enabled camera that's also doing cavity scans and making sure that everything's right in your mouth at the time of when you head to bed. Like this future is not like in theory, like there's every part of that like chain is things that like Robofo customers are working on and all parts of that. Now to give you a thing that is top of mind for folks of like the always-on camera thing and what is society gonna feel comfortable with that, I wanna give you like some of my direct thoughts. there as well. I think the transition of, um, even now folks— actually early on in smartphone territory, it made people uncomfortable that people always had cameras that they could capture other moments without people being aware that photos of them could have been captured. And frankly, even still, that's a real consideration of in public spaces capturing photos, not capturing photos. And over time, what society Which is ultimately the judge of this, is willing to accept. Is the benefit and life quality increase gonna be better than the new societal behavior? And I would take the bet that yes, because it'll start with simple things. Like think about my riding with my cycling glasses. Like they don't have a heads-up display yet, but I'd love to have turn-by-turn directions. And then pretty soon I'm used to like having that. small little display, and other folks are interested in that. I do hope that to build technology companies, you have to inherently be optimistic, because you're giving tools to people, and that means the tool will be used as a reflection of what you think about humanity. So if humanity is inherently good, then you're able to amplify those attributes. And I do think humanity is inherently good, even if there are bad actors. I think the same thing will be true for glasses and consent, and you can use prior technologies in pretty icky ways, The internet can be used to communicate with friends or support a small business online just as much as, I don't know, share photos that shouldn't be shared. And the same thing could be true the next generation of technologies. I have optimism that the benefits will continue to be things that folks will want to adopt, though the great news is, frankly, it's not up to me. It's like a jury of our peers of what others deem to be the case of where it's going to be useful and not useful. And then on the governance front, I also think it's important that we have systems and society and institutions that exist that also govern the use of these things in a way that reflects the preferences of people around us. There's a reason that privacy rights should continue to be strongly enforced and apply to the change of the times, right? The search and seizure was written well before the existence of cars and homes, and so what defines on rightful access and entry. And we should have the same sort of means-tested laws applied to new technologies for what is private, not private, public spaces, private spaces. And again, I remain optimistic that the principles we hold dear around being able to have a right to privacy and a right to using things the way we want to, at least in the country in which I live, is gonna be the way that the future products will be used and governed. So that's like how I think about it as like a participant in the system, just as much as someone that enables this future. But the good news is, man, the world is gonna get so much better. Like we have folks that are accelerating cancer research, cleaning up the world's oceans, removing pesticides from, from foods that we might produce, ensuring that electric vehicles are produced correctly, ensuring that stuff shows up at the same time. I like to joke that RoboFlow powers Santa Claus, and that future is happening now. And so I think that those are all things where it won't be without bad actors and its own set of like means testing sticky issues will happen, and there will be that case, that front page, that story. And we as a society will need to respond and ensure the frameworks and rights that we hold dear continue to be in place, even as the tools that we have to exist continue to be there. So that's how I think about that. And I think we have a responsibility to ensure that the future that we want to live in is one that we help foster in a lot of ways. I like to show examples of vision in our results that the vast majority of things is all about improving quality of life, not about maybe some of the bad implications or bad actors that things that folks might sometimes be concerned about. Those are some long-ranging thoughts, but a lot of folks ask me that question building the company that we get to build. So hopefully that gives you some color to how we've thought about it. Yeah, that's great.

That could be a good place to leave it. If I was going to ask one more follow-up question, because I sometimes can't help myself, it would be, do you think that there are technical either solutions or rules that we could define in terms of technology properties that would really help? And here I'm kind of thinking, because we've had a lot of this discussion about very general purpose models versus very specific, And I am struck increasingly by this notion of safety through narrowness, basically. And so from a bunch of different angles, I'm kind of wondering right now, is there a social contract to be had around AI that is like, we want to and we need to, and we all stand to benefit tremendously from solving very particular problems, but we also put ourselves at risk, perhaps, if we use fully general models everywhere to try to solve all these relatively narrow problems. So in the vision context, one that I could imagine is if you want to watch a public space for moments of violence or whatever, you could run that through a general purpose model that can tell you anything and I think in some places we're like identifying individuals by their gaits and their facial structure and whatnot. But an alternative would be like, let's have a very narrow violence detector model that doesn't really do much except sound an alarm when it has detected something that we want a higher order response to. So I wonder if you have any thoughts on, like, I don't know that One could argue, I guess, that maybe that sort of happens in some ways naturally because cost and efficiency kind of pulls things that direction. Somehow I don't feel that comfortable betting on that. And I kind of think we might need a little bit more of a social contract or some sort of idea of a new right. I'm always on the lookout for what are new rights that make sense in the AI world. And one of them might be to be classified by the smallest and most narrow purpose-built possible model for the task at hand, as opposed to being processed by some general purpose reasoner that could answer any and all questions about me. Anyway, I'd love to hear your thoughts on that.

Yeah, I spent some years in DC. I was an intern in the Senate once upon a time. Thinking about some of the institutional questions that affect this stuff is some things I've spent some time thinking about. My general thoughts is one of AI as a technology and firmly believing in the importance and value of open source for freedom of use and discovery and use case proliferation all stem from this idea of giving the rights to tinker, if you will, and like kind of use models where folks want to use them. And what does AI change in terms of how important societal rights need to take place. And where I come down is I think that the outcomes that we want to have in society should continue to be enforced, and AI is a tool by which those outcomes can be realized or not realized. In other words, to be really specific, it's like we have regulations that prevent fraud, we have regulations that prevent forms of violence or the actual kind of outcome by which something happens. A scammer could use an LLM to make it really easy to impersonate someone else, and they should be prosecuted for committing scam. They likely shouldn't be prosecuted for the size of model that they use, like the model they were using was too big or, or too small for a given example or task. So I think the idea of thinking about what's like minimal It's almost like minimal invasive use of minimal model size. I think where that gets into trouble is that the capabilities advance quickly enough or you have distillation that you end up having accidental corner cases where you catch too broad of a net that might stymie innovation and stymie adoption when in fact the goal was well-intended. I'll give you another great example. One could very reasonably steelman the idea that AI in healthcare has such far-reaching implications that there ought to be some form of, if you're going to use AI for patient health, then you ought to have a governance body approve or inspect or allow that type of use for AI with patient health. And someone could be like, very— that sounds like a very reasonable thought. intended position. And then I think about like users at RoboFlow, like this user at UNC Chapel Hill, who was using AI in their lab to automatically count the number of neutrophils that respond to a given experiment. And here you just have a lab postdoc student that's accelerating the rate at which they can experiment and doing a fairly menial, frustrating thing of counting. There's like hundreds of colonies of neutrophils that appear under this experiment. And how the proteins react, allow you to know if the experiment was good or bad, and if you do another round of treatment. And all of a sudden, that person that's just using AI in a fairly harmless, in fact, like quite useful way, would never endeavor to do that because it actually is touching patient health. And so you've caught yourself in this accidental position where you've got something that's well-intended. I don't want to harm patient health, or I don't want a model of a given size or a given use case When in reality, probably the way to attach that is you should be liable if you use procedures or things that— there's plenty of this already in the medical system— of you should be held accountable to practicing medicine correctly in the way that ensures patient health is respected. So I guess to be succinct, I wouldn't think that a narrow model size nails the way that You and I probably would want this technology to unfold. I do have optimism that types of regulations which inhibit misuse of any technology or of any behavior should be applied to AI, and that regulating at the tool level is rife for accidental slowdown and deceleration of what I view to be the modern industrial revolution that's gonna have consequential quality of life improvements in ways that we won't be able to fully forecast. And so one of the best ways to do that is to let it flourish and stamp out the places of where people engage in bad action. So that's a fairly general way. Of course, there's like individual things to think about, but that's how I've thought about where the field is today and where it shows a lot of promise.

Yeah, I think that makes a lot of sense as well. I definitely— another thing I kind of obsess about all the time is like, how do we avoid the nuclear outcome where we get all the weapons and don't get the energy? And I'm certainly not wanting to stumble my way into that sort of scenario. I think this has been great. Do you have anything else that I didn't ask you about that I should have, or anything else you want to leave people with before we break?

I don't think so. I really enjoyed the conversation. I appreciate the opportunity to chat about this, hear about some of the ways you've thought about visual AI, and almost give a refresh from CLIP 2021 to 2026 visual AI. And the rate at which this stuff moves, we could have a very different conversation 6 months from now about the same set of topics. So it's been fun to riff with you.

Likewise, looking forward to it. Joseph Nelson, CEO of RoboFlow, thank you for being part of the Cognitive Revolution.

Thanks for having me.

I know the cat beside the doorway, know the car beneath the rain. I know every edge and angle, every light and every lane. But the window has a border and the border has a wall behind it. I know the door. I see a shimmer of a world I've never called. Something pulls me like a signal, something just beyond the glass. Each time I think I've seen it all, the world gets wider fast. I know, I know, the center Where the same things repeat and answers line right up. But the edges, oh, the edges, where the shapes don't have a name, where the long tails wind like rivers. I've been dreaming I could claim the world so much bigger than this, than anything I've touched or known or traced. Every shimmer in the— The world is bigger than this. Periphery, every corner I have never faced. I want to run where the light is shifting, out past the frame, past the glass. The world's so much bigger than this, and I'm learning that fast. They built a palace out of language, taught the words to think in But a word has never watched a sunrise set a canyon wall aglow. You can read a thousand pages on the color of the sea, but the sea don't need a sentence, it just needs someone to see. A billion little moments no paragraph could hold. More shadows than a dictionary, more sky than can be told. The pendulum is swinging and I feel it in my chest. Something older, something Something faster, something opening it. Yes! A deeper way of knowing, rods and cones and ancient fire. A sense beneath the reasoning, before the tangled wire. Not slow deliberation, no, it's instant, it's alive. An eye that opened a million years ago and still survives. The world's so much bigger than this. From the rooftop to the factory floor. Every long tail winding outward. Every corner I've been hungry for. I wanna be where the morning is breaking. Where the real things cast their light. The world's so much bigger than this, and I'm wide-eyed in the light. What I dreamed about is finally happening. Every lens, every frame, every surface, every scene. What felt like slow motion is now accelerating. Doors swinging open to a world I've loved. But only through the window, only through the frame. The ones who built the window all these years are saying, "Welcome. We've been here all along." The world's so much bigger than this, and. I'm finally going to see it all, every long tail, every quiet corner. I can hear it, I can feel it call to see and to reason, to run where it's real. Past the window, past the frame, past the wall. The world's so much bigger than this, and I'm not going to No, I'm not gonna miss it. No, I'm done watching through the glass. The world's so much bigger than this, and I'm part of it at last.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website cognitiverevolution.ai or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, A network of podcasts, which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.