Adam Sand discusses Figure's robot training via headset data collection and his own Agentic OS that pays roofers to record their work. He walks through his custom software factory built on CMUX and Cloud Code, and describes an experiment running Astra agents that generated hundreds of pull requests but few actual merges.
What's going on, everybody? Today we are doing another live stream. Today what I am looking at is, uh, and well, I'm working on this, um, project that I have for our roofing company that is powering the Agentic operating system that will work for robots. And so I just thought it was kind of cool today, Figure, a company that I think is going to go very far. But I mean, I'm also excited about Tesla and others. I really— I just wish I got to see Optimus 3 more. But what I'm excited about is this Figure robot that has been doing— today a video came out where they did a time lapse of these robots cleaning houses, and they just took it to 30 different rental properties in the Bay Area and kind of just made a mess, and these robots went around and cleaned everything up. And it's kind of fun because You know, you really have to think like what goes into this, right? Like this robot here, they had to collect all the training data, which they're doing through a tool called Finger Index, where people are wearing these headsets and then they're collecting data. This is all over the world. Like, I think they've just crossed over 100,000 downloads of people wearing these headsets, and then they record themselves doing house cleaning and stuff like that. And then this company takes that training data and then turns it into the ability for the robots to do work. And so They kind of just put these— took these robots to these houses. The reason they did rentals is because it's kind of a zero shot, right? Like, they aren't trained, they don't know the area, they don't know the room. They just took these rental properties and then they went and kind of made a mess, threw these towels all over the place, made up and made a mess of the bed. And then these things actually kind of just go around the house doing their tasks, right? Folding the laundry, putting away the toys, putting the dishes in the dishwasher. Now Is it perfect? No, it could do, it could do better, right? But it's just, you can see here it's reacting to how they're, you know, like having the stuff tossed in the way, right? Like this is stuff that is all the little dynamic pieces that you're going to go into how robots are eventually going to be a part of our lives. And so if you look at where is the advantage, right, for the, the company like ourselves, right? We are building the system that is going to allow us to train robots to roof by collecting the data of roofers roofing. And we have a very special program in place to make sure that the roofers that actually record this training data get paid for it. Because if you, if you haven't seen, there's actually another cleaning company in San Francisco that is cleaning your house for free, right? So they will send robots to your house to clean your house completely free of charge, right? And the whole idea is that they're doing it to collect the training data, to get real-world training data, and then that is what they then communicate back to the central brain, right? You get this repository, you get this giant set of data that is used for training. And yeah, so this is how they train them, is using these Figure Index headsets, right? It's like their way of collecting the information. So then right now you can get paid to— you can get paid to wear one of these headsets and go around and actually collect the training data. And this is what allows you to actually take a robot and train it to do its job, which is very exciting stuff that this news came out today. So yeah, I don't know if you guys are excited about robots like I am, but that is what we're talking about today. Or at least that's not what we're talking about today, but that is what I'm talking about today. So I've just been really excited to see that kind of come into the real world. But if you were hopping on the live stream, I do these live streams at 9 o'clock at night for the pure purpose that I like to work on other things, and I just like to meet other like-minded people and connect. And so I'm always interested to see what you're up to, what you're doing. And, uh, and if you have advice, or if you think that you can tell me how to do what I'm doing better, I would love to hear it. And if I can show you something and how you can do it better, I'd love to hear that too. So I typically take these live streams as an opportunity to work, think through what I'm doing, talk about what I'm doing, and, uh, and hopefully find and meet some new people. Because I'm working it, I'm working on this stuff anyways. And so I've been lately, I've been kind of cranking out on this software factory, but I actually took a break from software factorying and tried something different. I wanted to see how Astra would do on Ultra. and actually just get it to run the machine. And so we're just kind of seeing, seeing how Ultra could do, and it did a pretty good job. Um, it worked on like— it ran like over 100 agents, right? And I gave it a few, like, you know, I gave it some constraints of how to work. I told it some things that it would have to— like, I gave it a— essentially, because I've been doing everything previously In CMUX, right, where this orchestrator would then run these other seats of Cloud Code. And I've been experimenting with, um, Pi Agent was in— which one was it in? No, did I close it? I closed it. Um, where Pi Agent was running DeepSeek, but then I was also running these ones, um, Cloud Code. I like Cloud Code because it was able to use the list agents and send message feature to send them back and forth. And so essentially the way that it runs is there's a ledger that keeps track of everything that I'm doing, and then there— and essentially what all the seats are doing reports back to the orchestrator, and the orchestrator is running its graph off of Linear, which you can see right here. And then Linear, essentially I've put everything into issues and tasks so that what can then happen is then it can fire up new seats take slices of work in accordance with the charter and then help me merge more work per day. So that's what I have been doing, but it kind of deviated and gone to say, well, I wanted to kind of see like what would happen if I gave the same charter, right, to Astra on Ultra and just gave it the job to do. And it actually did a pretty good job. The only thing I don't like is that it was too fast. And I call these waves, right? So you go on like wave 1, wave 2, wave 3. And so usually I just kind of run, run it in like, you know, it'll open up, uh, 5 seats at a time, it'll work through maybe, maybe 8 or 9 slices, and then close out, will merge a bunch. And then it's like, so I might get like you know, 3 merges an hour, 4 merges an hour, and then, um, and then eventually we'll do like, you know, say 16, and then we stop. And then it goes like wave B2, wave B3, wave B4. This one was kind of a tsunami. It did 29 PRs and we didn't merge anything. And I was curious to see how many merge conflicts there would be, and there is a few, right? There is a few. So the idea of running your own The idea of running your own graph is nicer, but I just wanted to see what would happen. And I actually gave it the job of like building out its own, uh, its own software factory. I even— but I mean, I've, I've done this before in that CMUX thing. I just wanted to see what would happen. And I mean, overall it's quite interesting. So if this is kind of the kind of thing that you dig, I would love to hear from you and see what you guys are doing. Um, and so if you guys are ever Like, if you guys are ever wanting to learn or connect, reach out, make a comment, like, share, ask a question, and I'd love to see what you guys are up to.
