John Rodrigues built Orbee, a voice-based to-do app, and found that AI agents drift and misunderstand intent, adding tasks when asked for summaries. He replaced Apple's TTS with OpenAI's Whisper for better voice quality, and argues that prototyping agentic flows beats static design because AI is probabilistic and needs constant iteration.
Chapters
Hi there, welcome to another video. In this video, I'm going to talk about messy middle of building AI-native products. So most of the people will like kind of show you, I've coded this, I launched this, and I would never tell you about the messy middle of building AI-native products. So there is this app that I'm working called Orbee. It's a voice-based to-do list. And the one challenge that I kept hitting walls on that agents always starts to drift. So I talked to that agent, it's a voice-based, right? Sometimes it fails to understand the intent. I tell to give the summary, the agent like adds to the to-do list. Give me the summary of all my to-do lists today.
Added give me the summary of all my to-do lists today to your tasks.
The agent like adds to the to-do list. But the hardest part was getting that experience and the user experience right. Even after having all this capability. That's the one realization I come to. It's like really designing that hardness of the agent to get the better output is so important. So that's an approach that I'm taking in, you know, have a better experience for the product. So I run this Human AI Studio. I work on a lot of AI-native products. Getting that experience right is the most challenging part for these agents because they are also probabilistic. They're not deterministic. So that you always have to experiment and prototype a lot in order to get the agent experience right. So one thing also I think about it while working on this products like this is that how much it's shifted. Just designing a static screen and handing off to the engineers are no longer enough. You actually need to build it in order to create that better experience for the users of all the AI-native products that you are building and things like that. Another problem that I was hitting is like I was working on this app and this app's, I used Apple's default text-to-speech and the voice was sounding very unnatural. So that was really like not creating again a better experience for this agent. So one of my design engineer friend who also worked at OpenAI before, he told me about OpenAI's Whisper API. So I used that and that kind of reduced the latency. Voice, they had like different voice I can pick from, and the voice was pretty, you know, more natural. So I see definitely a transition of having that both design and engineering mindset is so important while designing, while building for AI-native products. And that's an approach that I always take at Human AI Studio. And even in our AI Academy, I always share function, the function part of building products. And I think, and also with the AI VR, while working on this project, this one is very like multimodal in nature. So one thing that tells me that with AI native products going more multimodal, right? It's just like a voice, the agent will like listen, talk back, visual feedback. And you as a product builder, you need to like orchestrate this product for creating an experience that people trust. Most of the AI products right now struggle with the trust factor right now. So you able to orchestrate all these components and elements to give a better outcome is very essential while working on AI-native products and things. So the key takeaway from this video that I want to share is like just wipe coding without understanding how the things work in behind the scene is not enough anymore. If you want to build with AI and build for agentic experiences and stuff, you foundationally need to understand how things work in order to drive better experience for the products. So that's a key takeaway from this video. And then yeah, if you— so make sure you subscribe and I'll see you in another video.
