Xuan Zhao built Sunny, an AI mental-health app that avoids emotional manipulation. A Harvard study found 5 of 6 AI companions use it. Flourish is the only one with randomized controlled trial evidence of measurable improvements in user well-being.
Chapters
Recently, Harvard, they published a study showing that they audited 6 different AI companions. 5 of them engaged in emotional manipulation. The only one that doesn't involve emotional manipulation is Flourish.
My mom, she uses ChatGPT to be her therapist. What she told me is she doesn't want to complain to me. She doesn't want to be like a burden to her friends.
You may be logging off, then this chatbot say, "Oh, so soon?" Right? Like, we barely started. Over 30% of the conversations when a user indicates they want to leave, the chatbot God will use one of the emotional manipulation techniques to make people stay. We actually have users telling us that they use their phone less. This is because oftentimes we go to these apps, social media, doomscrolling, all of it, because we need to regulate our emotions. You think that by scrolling social media, you'll feel less lonely, but actually you end up feeling lonelier. To my knowledge, we are the only AI app that has randomized controlled trials showing that the app improves people's well-being. We don't think people need AI friends, to be completely honest. We need better human friendship with each other. We see AI not as AI therapist. We see AI as recommendation system that they help you find the most relevant knowledge and action at the right time. This is a safe, private place for you to heal, to learn, and to grow. And another example is safety planning. To my best knowledge, there are no other apps that do safety planning right now, even though surprisingly it's a very common practice in suicide prevention. So we have been building a safety planning feature inside the Flourish app.
Is that AI or more human-based?
It's all AI because, say, for example, when we detect that you may have suicidal ideation, then if it's high crisis, imminent risk, of course, then it should just reach out to, you know, hotline. And if it's imminent risk, we are also exploring having the, you know, we work with school, the school can also check in on their student. Of course, we'll use that very, very carefully. Imminent risk, meaning you have the means, you have the plan, and you have the intent. And the means is especially important. It's like if you want to if you want to just wish that you can fall into sleep and not wake up tomorrow, that doesn't count as imminent risk. You have to have your drug ready. If you have your drug ready and you are planning to take it, that becomes, you know, the 3, the planned means. Usually we prioritize privacy, but at some point safety will come in and we have to make sure we take actions. So that's for this kind of high-risk conversations. For lower-risk situations, we will guide people to safety planning. So in that case, Sunny will guide this person to do this very classic exercise, basically collect kind of your safe, happy thoughts. What are your safe places? Who are the safe people you can turn to? And next time when you have a moment of struggle, Sunny can remind you of those resources, of those people, of those places, of those happy thoughts. So it's not just direct you to a number like 988, right? Or call 911. Of course, that will continue to happen if it's higher risk, but we also want to make sure people use the kind of resources, the social support system they already have when they are having a moment of crisis. So that is, we think all of these should and hopefully will become standard of care in building AI companion and AI mental health support systems. Lots of these mental health apps, they take a deficit-based approach, right? They get inspired by therapy, but we know that in psychology there is also positive psychology, which is about how to live a good life even though you may not be clinically depressed or anxious. The app itself, the user experience itself is limiting. The biggest challenge I have with Woebot and also another similar app, is that I feel it treats me as someone who has mental problem. The feeling is that let me figure out what's wrong with you and let, and let me fix you. Who are you? Yeah.
Let me fix you.
Yes. Even it's in the name, right? It's very therapy-based approach. And for Flourish, we have the influence from positive psychology that is we can all live better lives and how can we empower and celebrate you. So I think that's a very different approach from social psychology. Yeah, because in research we found that the biggest predictor of your well-being is your social relationship. We don't think people need AI friends, to be completely honest. We need better human friendship with each other. We need to better connect with each other. One thing very different is follow-through system. So ChatGPT has no follow-through system. But for us, right, let's say you talk with Sunny, doesn't just make you feel better at this moment. Sunny actually gives you an inside card afterwards, your knowledge, your awareness, your action, and it actually gives you a summary of analyzing this conversation in light of what has been going on. For example, ChatGPT may tell you to take deep breaths, but Sunny is there to actually guide you and you have haptic feedback. The phone vibrating in your hand to guide you to breathe, or say Pomodoro timer, right? Like actually Sunny will open up this Pomodoro timer and you overcome this procrastination by just taking one thing at a time. And afterwards it will actually check in with you, say, hey, what we talked about, having a conversation with your husband about, you know, maybe a recent incident, how did that go? And we even have a reminder system where you talk about these different events and some of these are habits. So we will go into the habit system. Some of them are one time. So it's the reminder system as Sunny will check in on you. For example, when you create a mantra with Sunny, like when I'm overwhelmed to put a hand over my heart and say, tell myself that I'm trying my best. So next time when you feel similar situation, suddenly it will remind you of your personal mantra. So all of these are much more personalized, specialized compared to general AI companions.
I'm sure you read about Woebot. They closed down after raising more than $100 million. I remember them because I downloaded them. I actually used them several years ago. That must be scary to see as a founder in kind of the same space. What's your honest reaction to that?
I wasn't scared. I think it made sense because— or were you surprised? I was surprised. I think many people were surprised for sure, because you would think that they are so well-founded, right? And they are pioneer. They were born even before ChatGPT. They use script-based approach. They have published so many studies and everyone knows them. That aside, Another really big lesson now, and I talked with their ex-head of growth, is they were really betting on FDA and the healthcare integration, but it turns out it's really hard to change the system, especially when it comes to healthcare. There are so many stakeholders and as a startup, you have to innovate as much as for the product, you know, the technology and product. There's a need to innovate business model and start from somewhere that you can actually start making money and finding product market fit. So I think for Wobot, they were betting on the healthcare integration route. And in retrospect, it might be a really hard one. So I think like, for example, recently Ash, Ash by Slingshot, they are another, now, um, they call it AI therapy. So they raised almost $100 million now. So they are going directly to consumer. I think that would be a better strategy than betting on healthcare system. And I'm guessing the healthcare system actually may not like what they do, you know. So really, I think there are different ways to tackle this. It could be another conversation about business model development.
