A Keyboard-Less Future: Reinventing a 150-Year-Old Interface with Wispr Flow

The time between when they see the dictation show up to pressing enter, it's half a second. Most people don't even read what Flow produces anymore because they're like, it's probably fine. It did a better job than I would have anywhere.

What I care about as a user is getting to hit send as quickly as possible.

Our mission is to make interacting with technology feel just as effortless and natural as talking to a close friend.

Hi, I'm Chelsea Taylor, and today I'm excited to be joined by my colleague Hans Tung and sit down with the co-founders of Whisper, Tanay Kothari Kothari and Sahed Garg. Whisper is tackling one of the most ambitious challenges in technology, reimagining the primary interface between humans and machines. Their first product, Whisper Flow, is seamless speech-to-text in every application on your iPhone or computer. At Notable, we're so excited to back Whisper and recently had the opportunity to lead their Series A extension round. Today, we'll be sitting down with Tanay and Sahed to dive into the behind-the-scenes story of building Whisper and a future of a voice-powered world.

Well, gentlemen, thank you for being on the show. You guys were college roommates at Stanford. Tell us how— what was the early days like? How did you guys meet in the same dorm? And then how did you guys decide that you want to do something together?

Yeah, so Tanay and I actually met literally on the first day of our freshman dorm. So we were both in Donner at Stanford, great community of former founders from there as well. We instantly clicked because we both had so many of the same interests. We loved math, we love computer science, we would stay up on weekends taking extra graduate courses just for fun. Graduate courses? Yeah, just for fun.

He was the only person I could go to and be like, "Dude, I'm really sad. I got an A and not an A+." And he would get me.

I think the hardest part about being friends, I think we actually both gave each other so much imposter syndrome. Being at Stanford. And in many ways that's telling even now, right? We both learned so much from working with each other in so many different ways. We thought we were really similar people freshman year, and then we ended up rooming together sophomore year and we just realized how different we were, but how many interests we shared. Like both of us started dating our partners in high school. We've been together for 10 years. Kind of crazy to see those kinds of similarities as well. Which really just strengthened the friendship over the next 3 years.

How do you guys complement each other?

I am a very impulsive decision maker. Sahaj is way more thoughtful, which is great, 'cause in the company, like sometimes you need to like make decisions quickly. Sometimes you need to think very deeply about stuff before going ahead. So we balance each other out. Sahaj is very focused on more core research, pushing the limits of technology, which really adds to a lot that we do Whisper side. And my head goes a lot towards marketing, growth, product, etc. And so I spend a lot of my time there. So even though we both have a technical background, it's the exact opposite kind of technical background that you can imagine.

The way I think of it is there's two parts of running a company. One part is saying yes and the other part is saying no. And we both complement each other because he's the guy who will say yes, I'll say no to instinctively. And on the flip side, like, I rein us back in and make sure we focus on the half of the things and do those things things in a really incredible way.

Yeah.

How many iterations did it take you guys to get to the first idea? Yes, you want to build something together?

You know, initially when we were in college, we'd wanted to do things together, so we would work on a bunch of projects and like stay up till 4 AM every night building something new. And then Saj wanted to do a PhD and I wanted to build software companies, and we were like, this is completely different, we'll never start anything together. And then one day in 2021, Saish calls me and he's like, "Tanay, I'm leaving my job. I'm not doing a PhD. I wanna start a company." And at the time I had a job and I was like, "You know what?

Doesn't matter." It doesn't matter.

This is an opportunity of a lifetime. I wanna do something together with this guy. And so we started and funnily, the current idea for Whisper and the vision behind it was the first thing that I talked about or that we talked about with each other. And then we were like, okay, let's like not jump in so quickly. And so we actually went through about 100 other ideas.

100 other ideas.

100 other ideas of like different verticals and spaces and problems in the world that we wanted to solve. And eventually we just narrowed down on Whisper. It just felt right.

What clicked for you that, you know what, I'm gonna not pursue PhD anymore and I would wanna build something and I'm gonna go call?

Tanay Kothari. The thing that I really love is solving really hard problems that make impact in people's lives very directly and being very problem motivated. And with the role that I had before Whisper, I was running the AI team at a startup called Luminous Computing. There we were building optical computing hardware for ML acceleration using light to accelerate data movement computation. Crazy shit technically. And the thing I loved about that was figuring out what problems it solved for people in the ML community and designing the computer from the ground up to actually solve those problems. And as a result of that, I really wanted to focus on industry research where we could take problems that we knew mattered, devote all our energy towards solving those problems, shipping them to users, making sure that they were actually the things that people need, and then iterating from there.

It's been really great to hear about your guys' story of how you meet freshman year and then you come today to build Whisper. But Tanay, I know you also have another interesting story Whisper is not the first company you've built. You actually built a company when you were younger, maybe 13.

Yeah.

That you described that Google maybe shut down. Maybe tell us a little bit about that story and how is that maybe formative for you as you think about being a CEO now?

So this was when I first started to learn how to code. I was 9, 10 years old and I wanted to build Jarvis from Iron Man. Mm. And so me and a friend of mine built one of the world's first voice assistants. And at that time, I also loved music. And so if you remember, LimeWire was really popular then. That's the best way you could download music without paying 99 cents to iTunes. And when LimeWire shut down, it was painful to download music. So we added a thing where you could say stuff like, hey, play me the latest song by Imagine Dragons. And it would download, find that song in the internet, download that for you. But one thing we added was if it didn't find the MP3 on like MP3scol or one of the other websites, It would actually go to YouTube, find the video, convert it to an MP3, and then download that for you. So Google was like, why are we getting so many requests from this corrupt server? And they actually like sent us a notice. I was terrified. I showed it to my dad. He didn't even know I was doing anything like that because I was— I would only code at night after he'd gone to bed. And I shut it down. And initially I was just really mad and also really sad. Like, I'd spent 2 years building this thing. We had gotten to 2.5 million users in just 6 months, and now this whole thing was dead. But after a couple of days, it just hit me that they just killed my app. I can still do whatever I want. And so I started building something else again. And to me, That essentially what it did was I felt less personally related to the things that I was building. So if the world or anything at some point was like kill something or some of the work I did went to waste, it's like, I'm still here, I can still do more things and keep going. 'Cause that happened a lot over the years. That happened multiple times with Whisper when the world just said like, nah, you're not doing this thing anymore. And we realized like, hey, I'm here. We got some great engineers.

I got this guy.

Uh, we could just continue building the things that we think are, are right with whatever we've learned now.

Yeah. I mean, that's a super important lesson. And I think it's part of also the story, like you just alluded to for Whisperflow, um, and where you guys are today. We obviously know you as the best-in-class voice dictation, um, product in the market, but it started. In a very different way. Maybe tell us about that kind of pivot story. And again, you alluded to this has happened to you before, so you weren't scared to kind of switch and dive in again.

Oh, I was terrified.

Okay.

To clarify, you were scared.

Yes. Our goal since the beginning for Whisper has been to solve the same problem, which is that we want interacting with our devices to feel like talking to a friend. We want to be stuck less behind screens and have more and more natural interfaces. And we started the company thinking the problem with that was that people wouldn't use voice around other people. Think about how awkward it is to use Siri when you're around others. And so we started by building a wearable brain-computer interface that could understand what you were thinking silently to yourself.

You had a hardware background, so it was easy.

I had a hardware background. We'd done machine learning work. And so it was a great fit for the things that Tanay and I knew how to do. And fast forward a couple of years, the R&D for that was going extremely well. We actually hooked up the prototype to all the voice interfaces that existed then, like Siri, Alexa, and ChatGPT. And we found that none of us wanted to use our own product because we hardly even use those interfaces even when we were alone in a closed room. We weren't going to wear a wearable right now for those kinds of interactions. And so we started pushing ourselves internally to build software for the hardware, the operating system for voice, for interaction that would actually make it to that we would want to interact in a more natural way. And that eventually turned into WhisperFlow. And we realized just how much there was to do there to make that take off. And I think the lesson for us there was the market wasn't ready for solving the problem of using voice everywhere at that time. Mm-hmm.

Right.

And the moment the market's ready for that, which is actually going to be pretty soon, there are going to be really interesting things that we need to build to make that possible. But we'll know exactly how to solve that problem because we know what the user's pain point is. We see them, we see how they're trying to do it right now, and we can help them fix that. But it's been a wild ride.

So you're saying we're going to go from voice dictation to voice controlled or voice managed services? Yeah.

Very soon.

Very soon.

It's what Siri should have always been. Yeah.

Figuring out what users would be willing to do is such a hard problem to solve. How did you decide and stumble onto that Whisper flow was the right UE or UI to make it work? What was that 0 to 1 journey like?

So we actually started with a lot of things that didn't work. So we started by building a lot of really well-functioning voice agent or voice assistant-like applications. This is like right around when ChatGPT was about taking off, and you could do things like say, hey, like find me dinner with Hans nearby and invite him. And it would actually go do the search, book a reservation, send you the invite, do all those things. And what happened was we only ever tested it internally. But like in practice, I get dinner with you like at most once a week, you know. That's a good amount. And that would be a delightful week for me, you know. That's a frequent amount. And so we were thinking about, hey, what's the interaction that we actually do the most all the time? And hey, what we do all the time is we communicate. We type and we communicate. And so that's the moment where we started building WhisperFlow, the version of AI-enabled voice dictation. And for me at least, the moment where I got a lot of conviction in it was when I fell in love with the product myself. Because despite being a startup founder and not somebody who falls in love with startup products very easily, like I'm not an early adopter of a lot of tech, and I fell in love and I was like, wow, if this is how quickly this can shift my behavior, If we can round out all the edges, make it so that it actually gets you flawlessly every time, make it so that it can start to take the actions on your behalf that you actually want to take and have them work every time instead of work 50% of the time, it will completely transform for me. That's my side of it. Anything else for you that kind of got you to feel that magic moment?

The thing that was shocking to me, and so we were working on Whisperflow while we were building the hardware as well. You were building the hardware so that people could use voice around everybody else. And I just vividly remember one day walking into the office and I see half our team with these mics. So we've had these, the gooseneck mics for a while. They're on their table using these mics to talk to their computers. And I look at this office and it just hit me like, there is no problem to solve here necessarily. That's that. Major. We're solving most of it with this software and a $10 mic that we bought, right? And that to me was the first time I had seen us build something that actually creates behavior change. And then diving into like what created the behavior change, the biggest thing was like how frequently do you do a particular action. You might like book a flight once a month, uh, but you type hundreds of times a day. So if that is the thing that we can hook on to, that is like people use Whisper hundreds of times a day, it's bound to build incredible habit, it's bound to change human behavior. And seeing that initially happen with the first group of 10-15 people was honestly what Sahaj and I bet on at that time. This is now barely 1.5 years ago. That I think there might be something here. And we pivoted the whole company to just focus on that.

I remember the first time you told me that story, you remind me of Slack. Our startup as a game studio to build games, next generation games across 4 or 5 different locations. So it needed something like a Slack to communicate. And ends up being that the studio didn't go anywhere, but the communication tool is something that everybody used. All the time and that become a new version, the new product, a new company.

Yeah, the Slack, I think the Slack analogy makes a ton of sense. And something that we also read when we were thinking and we're really excited about partnering with you guys is something you wrote on Product Hunt actually, and you talked a lot about this pivot story. But one thing that really resonated with us is you said not only was kind of your pivot a focus on changing the product, but it was also it pushed you to change the way you make decisions at the company. And you already alluded to this earlier, you complement each other very well. Maybe one person saying yes while the other is saying no. But maybe elaborate on that a little bit more. How has this, you know, pivot or the new iteration of what Whisperflow is, how has that kind of inspired the way that you lead the organization and the company?

There's a bunch of different ways. Two that are the biggest for me is the first is starting with problems that we see people experiencing and that we see people experiencing frequently. So that's the root to us for all behavior change. You get people who are doing something, they're going about their lives, they're busy, they can't think too much about learning a new tool. They just want to change one behavior that they're doing that fits right into what they already experience as a pain point and make their life better. And it feels small to start. To only start by changing one tiny thing that's frequent. But if you do that, then we get the opportunity to build the second and the third and the fourth and the fifth. And all of a sudden it creates this ecosystem of behavior change. And so, you know, the hardware was a big bet on changing people's behavior dramatically. Right. Right. Have them adopt a new wearable, learn a new way of, you know, thinking silently to themselves, then adopt voice, then do all of these kinds of things. And that's too much to ask of a person. All at once. There's that half of it. I think the other half of it is also when it comes to building new product surfaces that really stick, it's a little bit of humility around what we know and what we don't know. A lot of times you have to just build it, try it, feel it yourself, and see what feels magical and what doesn't, and be willing to acknowledge those hard truths to yourself. And so you can spend all the time that you want to thinking about something or designing something perfect. But in reality, just shipping a quick experiment as fast as possible so you can learn is the most valuable thing that we can do.

I think a lot of founders try to be the next Steve Jobs or the next Elon Musk and try to do things that are really grand. Most of the time they don't work out. Being focused on changing one behavior at a time, but it's a high-frequency kind of behavior, is the best way to go about Building something new that can last.

I think the thing that I appreciate now is I actually think Steve Jobs and Elon actually only sometimes change one behavior at a time. Like, if you think about the iPhone, the problem it solved initially is I was carrying around an iPod and a phone in my pocket, and it just put them in one device. That's like—

without iPod, it may not be an iPhone.

Yeah, and the App Store wasn't even there until iPhone 3. Like, that wasn't something that the iPhone came out with. It changed our behavior one step at a time, starting with something small. Even Tesla. Okay, I'm going to take a vehicle and it's going to be electric, but I know how to drive a car and it's going to be a car and it will solve that problem really, really well. And there's a tremendous amount of work that goes into solving a simple problem. Not to diminish that, but like they both actually built incredible products by distilling things down to their essence and only changing a small thing in a person's life.

Something that I appreciate that you said is voice dictation on its surface may seem simple, but we both know, and we've learned a ton from you guys over the last couple of months, like this is a really complex technical challenge. So be curious to know kind of how you think of it holistically to tackle this problem.

So the way we think about it is, is pretty simple. What I care about as a user is getting to hit send as quickly as possible. And so for typing, you know, I can type whatever I want to. For voice, that includes getting the results back, and how long it takes for me to fix all the mistakes. So what we track there is what we call zero-edit dictation. What fraction of your dictations come out perfect on the first try, so you don't actually have to fix anything? And that means like us going on your behalf and fixing when you correct yourself, when you ramble for a while, when you have run-on sentences, um, when you might not be structured in what you're saying. And What we do is we constantly work on iterating here. There's a bunch of things that we have learned that I think are different from how other people are approaching it. So most people have approached voice as, hey, let's deploy smaller and smaller models on the edge, on your phone. But what happens there is then it makes more mistakes, then you spend more time correcting them, and it's not actually worth it. On the flip side, what we're trying to do is actually deploy bigger and bigger and bigger models. As we train our own speech foundation models. And the power of that is if it never makes a mistake, the end-to-end user experience is so much better. And here we're measuring it in all sorts of hard conditions, you know, with people whispering into the microphones in the office, when people are walking on a busy street and there are loud cars going by, all of those kinds of settings. And the core technical challenges are let's train these models to actually work in real-world settings. They work great on the benchmarks, on all these offline statistics where people are like 99% and none of us use it and feel that ourselves. And then the second is making the inference of these feel instantaneous. It's got to be in the cloud, but it has to feel so, so fast. And so it's this constant tug of war between train a bigger model, make it faster, train a bigger model, make it faster. And that's the magic that will make it so that voice becomes something that people can trust.

At this point, for kind of top users or folks that are using WhisperFlow a lot, what has been kind of their behavioral shift in terms of, for example, how quickly they press send after utilizing the dictation?

Usually, the median person, the time between when they see the dictation show up to pressing enter, It's half a second.

Wow.

Half a second.

Half a second.

Impressive.

Most people don't even read what Flow produces anymore. 'Cause they're like, it's probably fine. It did a better job than I would have anywhere. And that to us was shocking. And also the thing that we really, 'cause we expected people to trust Whisper, but not so quickly. And so that's what we push for more and more. And so internally we're looking at, hey, who are the kinds of people? What are they doing? Where it's taking them longer, where they're, reading the text, which means they haven't built the trust yet. And so I think a lot about what we're building at Whisper is gaining the user's trust, because that is incredibly hard for any product to win over. But once you have that, you can do so much more for that person, with that person. And, um, that is even harder for us because we decided not to just build this for Silicon Valley techies, but just the average person out there. The thing is like my dad is never gonna write a system prompt. My grandfather doesn't know what a prompt is. Like most LLMs and these AI tools make you configure so much beforehand. We actually said, hey, you know what? We don't need you to tell us how you write. We don't need you to teach us like Dragon Dictation, how you say each word and the names that are important to you. What we're going to do is we're just going to learn from you just like a person would. We're just going to be this like new person who's joined, who's with you side by side. And that actually adds so much more complexity to the technology that we're building. But the benefit we get from that is it's not just the few million tech-savvy folks in the world who use Whisper. It's actually accessible to billions of people around the world. Who can just get started and press one button and get flowing.

It's because communication is just so core to everything we do, right? I've seen this with so many people. The better and the more effectively they're able to communicate, it just completely transforms the impact they can have in their career or the level of connectedness they can feel with their family and their friends. And so being able to do something that unlocks that ability for people who otherwise have barriers to communication is tremendous. And then even for people who don't have barriers to communication, just making it more and more frictionless allows people to be more connected.

I think I shared this when we were getting to know each other, that I had gotten my family to start using Whisperflow. And in particular, my mom loves Whisperflow now. And similar to what you alluded to with your dad, like, she's not someone who's going to be learning kind of the most robust ways to build like an AI agent, but she now can communicate much quicker with me, with my brother, and it's like, breakthrough. But then additionally, I don't know, I want to ask you a little bit about use cases. But similarly, I was, um, over the weekend was doing some work with my partner and he's an architect and he uses Whisperflow to actually edit, um, drawings like architectural drawings. Yeah. So you should talk to him about how he does it, but he's obsessed with Whisperflow because a lot of what he's doing is dictating and describing how he wants a drawing to look. And so he'll actually dictate that into his into Revit, which I won't go too deep into architectural softwares, but, um, he's using it too. So it proved to me through my own kind of learning, I can use it for email, but then this entire different ecosystem, um, can also use, use this software really effectively. Um, those are my examples of use cases, but I love to hear from you both. What are some of the most surprising use cases you've seen kind of come from Whisperflow users?

The one I remember is Uh, this guy, he's one of my favorite guys, uh, Greg. So Greg messaged me one day and he was like, dude, you know what would be cool if I wrote a book? And at this time, like Whisper had 100 users, right? And he was like, I'm gonna write a book with Whisper. I was like, I don't know how you're gonna do it. Like we just support 5-minute dictations. And then a month later. He sends me a manuscript and he actually published that book. He published another book a few months later, and we were spending time with him and learned actually so much about how convoluted and hard it is to be a writer and work with editors and publish books and how much strain it puts on their lives. And the fact that they can just be so much more creative.

Uh, it's easier just to say something, they actually write something.

Yeah, he's so much easier, you know, say things. He used it a lot to edit his book, and I think he's writing a third one now. Like, he's, he's on a roll. He's like publishing book after book. He's like, I would write one book every two years, and I'm writing one book every two months, uh, which was huge. And the other really fun one that happened recently is there was this guy who was who ran a marathon. I saw this while wipe coding an app. He had a backpack on his computer. He was wearing AR glasses and just using Whisper to talk to it. And he actually got a pretty good time, like better than, than mine or Sahed's marathon as well. Um, and so I'm just, uh, people surprise me all the time.

You don't have to squint that hard. You can see how this could be the beginning of a new operating system or user interface that changed lives.

Yeah.

No, genuinely. And I think on that, as we build this more and more, I still talk to a lot of users every day. And what I realized is there are 10 million people in the world who have Parkinson's. Their hands actually shake, so they can't type on their phone or computer. Whisper for the first time lets them use technology because keyboards actually are a big obstacle to using anything, right? Uh, there are 700 million people with dyslexia, and we literally tell them like, hey, if you want to use technology, you need to literally spell words, which is the genuine hardest thing that is there for them in, in life. That's 10% of the world's population. And so for all of these people, Whisper just ends up leveling the play field and removing discrimination that keyboards, which looks Nokia actually have been doing for the last 150 years. Um, and so, so those messages to me are, are so heartwarming when I get those from people, because yes, like, for some people it makes you really productive, saves you hours, but so for so many more, it lets them do something they could never before.

We spend a lot of time talking about how big the opportunity is just today with the product that you have, which is voice dictation. It levels the playing field, it allows people to be more creative, it allows people to be more productive. Um, but we've all know that you guys have a bigger vision. Talk to us a little bit, like what happens next for Whisperflow, or what are you most excited about?

Our mission is to make interacting with technology feel just as effortless and natural as talking to a close friend. And to us, the first thing that we needed to do was crack voice input. Make it something so you can just blindly trust a system that I will say things and it will understand things from me. That ship product now has a cult consumer following. And that gets us to the next part, which I've been really excited for for years, which is Whisper, not just writing what you say, but actually doing things for you.

Yes.

And helping you proactively. These two things are really important because people start using Whisper everywhere on their computer and now they're expecting it to do more. There's so many things that we learn, or Whisper learns, about the user as they use it for a while. And there are incredible number of opportunities to help them in places to save time, make them better at everything that they're trying to achieve. And the way I think about how we prioritize doing these things is, number one, whenever I'm talking to users, I ask them to think about what is work in their life that feels like grunt work. That they would rather not spend time doing so they have time to do what really matters, that gives them energy, that gets them excited, that gets their creativity out. And so you take all the grunt work, and those are the things that I want to start automating away with Whisper so that it— more and more of their life just happens through Whisper, and they have, they have time left, uh, to do everything else. And so that is what the next phase of the company looks like, and a lot of what we're going to be shipping next year. And the really important thing here that I— it keeps me up at night a lot is there are so many companies that have tried to build this. So many companies, like earlier they used to be called assistants, now they're called agents and, uh, agentic workflows. And it's all at the end game of being able to do tasks on behalf of people. And the biggest thing that I want to ensure we deliver to users is reliability. That if Whisper tells you it can do something, it actually can. The counterexample to me that I always think about, the failure case for this is Siri, where it promised 1,000 things, it can do 50, and it can maybe do 2 really well, which is why we all just use it to set timers and alarms. And that is what happens when you focus on building capabilities and promising a lot more. Without building the reliability underneath it. That's the thing that made Whisper Flow really successful, and that's something we really, really, really need to focus on for this next phase of Whisper.

If I think about that from a user's perspective, right? I use voice because it's faster than typing. But if I have to go clean up its mistakes when it takes an action on my behalf, like if I said, hey, message Hans and Chelsea saying that I had a great time at this podcast recording. And it messages some other Hans, I will be so furious. Right. And so everything here is about figuring out how we actually build the voice agent interactions for the consumer that feel better, right? If I'm interacting in real time with it, it has to be so fast. So I'm not sitting and waiting for my AI browser for 15 seconds for it to click the wrong button on the screen for me, which is the experience that I have today. Like, 15 seconds and it'll hit send before the email was actually written correctly.

Right.

And so very, very focused on that. And I think one of the key ways that we can actually fit, fit right in here is, so there's right now, which is voice-to-text. Then there's kind of next phase, which is, hey, how do we go to helping you complete certain tasks? So for example, all throughout the day, I have all sorts of thoughts going through my mind. And I can't solve them all at once, so I need to queue them off to somewhere. And so I can jot them down as a note to myself. And what if every time I did that using Whisper, I don't have to context switch anywhere, right? I can do that from anywhere on my mobile or my computer, goes to my Whisper notes and it kicks off some work in the background. So when I come back, I have something that I can work off of. If I'm planning a trip, I actually don't want an agent to plan my whole trip for me because I have preferences at the end of the day. But if I say, hey, I need to plan something for my trip, and then when I come back, I'm like, okay, great, here are like 8 suggestions of hotels, here are some ideas for these things. Now I'm already kickstarted, right, on my process. And based on that, we can figure out, hey, which of these things can we do really well, really interactively for the user? Then you can start to do that in real time, right? Support all those kinds of interactions that people want all the time. And we hear this, for example, for You know, small things that are huge quality of life changes, like send a Slack message without opening Slack and without seeing the barrage of 50 notifications that suck you away from whatever you were focused on in the first place.

Yeah.

And so there are so many of these interactions that we hear people asking for day in and day out. And if we can do them well and execute them well in a real-time user interactive environment, right? So it feels good. Then it's going to completely change what my interaction with my device looks like.

Amazing. Thank you guys so much. This was really fun.

Thanks for listening to Notable Perspectives, a podcast from the team at Notable Capital. If you enjoyed the show, be sure to give us a rating and subscribe on your favorite podcast app. You can learn more about Notable Capital by visiting notablecap.com.