Jason Yuan - Going beyond craft to design AI products with soul (Dive Club Ep. 74)

I think the best designers maintain a sense of play, and you can tell in their work. Very high levels of craftsmanship. That's a given. You have to be good at the execution, but then the most important thing is, like, having soul. I think there's lots of really great craftsmen, but there are very few people who you meet, and they're trying to say something through their work that's more than just, I made a beautiful, polished thing. And usually, I think with, like, really soulful artists and designers, like, there's something that's driving them, a story they want to tell, a thing they want to change. And, like, that, I think, is really important in making anything profoundly human centered.

Welcome to Dive Club. My name is Red, and this is where designers never stop learning. This week's episode is with Jason Yuen, who you might know from his project, Mercury OS or maybe his time designing at Apple. But the reason that I was so excited to have this conversation is I've recently fallen in love with his new product called.it's by far the best personal AI that I've used. So I wanted to get the full backstory story. In this episode, we talk about everything that went into designing the product, what Jason's looking for in a new design hire, and he even shares some pretty spicy takes about the future of generative interfaces. But I wanted to start this conversation with a deep dive into how Jason thinks about storytelling, because the DOT websites feature some of the most compelling narratives that I've ever seen.

We worked with this guy called Joe Sal. He worked with, like, yay. He did skims. He worked with Airbnb Open. He's amazing. It started as a simple branding exercise. We were trying to think of, like, the identity of new computer, and we just wrote poetry from poetry. We started writing stories of, like, the role it will play for people, or even stories of the future where we think life would look very different. And Joe wrote this, like, really moving short story about this character named May. And it was so moving. I, like, I felt like I was about to cry. And the original draft was, like, super long. It felt like I was reading a screenplay or something. And when we were thinking of announcing and launching Dot, I'm just so tired of. I just. I just feel like everyone's doing the same thing over and over again using the same references. I mean, all the bento grid stuff is basically just, like, plagiarized Apple Marcon. Unless you're announcing an iPhone. Like, why are you doing that? Like, can you, like, go touch grass? Like, go outside like, literally go to any museum. Go, like, roll around in Central Park. I don't know. So I definitely didn't want to do that. And also, we have a bit of a problem, which is the her effect, which is there's so much literature and science fiction about AI and what AI assistants look like. They watch her, they watch Iron man, they see Jarvis, they watch Megan, and then they're like, oh, that AI assistant. That's what it does. So we can't just be like, oh, it's a personal AI companion, because everyone already thinks they know what it is because of the stories that we've consumed, that we've read and watched. So we were like, we need to move people. We don't need to sell the product. We just need to move people. We need to give people a story that will help them feel something for the character Mei and relate to her in the same way that when you watch Hamlet, you're relating to the grief and the chaos and the torture. So it's about finding that universal human truth and extracting it and then presenting it and just going like, it's a story. Deal with it.

Look, I tried, but, man, I couldn't get excited about midjourney. But then Visual Electric launched, and it totally changed the game for me. It's image generation, but on an infinite canvas. So it feels so natural. I use it for absolutely everything. Everything. Now I even have a custom style preset that I use for all of the COVID photos for Dive Club. The whole product is super intuitive and very clearly built specifically for designers. So the next time that you need imagery or mockups or maybe you just want to get a little bit creative, I cannot recommend Visual Electric enough. Head to Dive Club, Slash Electric to try it out. And if you use the code Dive Club, you can get your first month for free. That's Dive Club slash electric. So I recently did an episode with Noam Seagal, who's a longtime research leader at Airbnb, Wealthfront, Twitter, and he kept raving about this new product called genway that he was using in his current role at upwork. Now, it uses AI powered interviews to help you gather really rich qualitative data at basically an infinite scale. And I tried it out. I liked the product so much that I decided to use genway to gather feedback from people listening to the show. So if you have a few minutes, it's kind of fun, Head to Dive Club interview, and you can get a little taste of what the future of research looks like. That's Dive Club interview. Okay, now on with the episode. It's interesting to hear you talk about, like, reading a screenplay and Hamlet because you attribute a lot of your abilities as a storyteller to your background in theater. So, yeah, can you talk a little bit more about, like, what makes that part of who you are as a designer so valuable?

Oh, my God. It's because I love attention. I mean, no, it's because, okay, like, the ancient Greeks did theater because they wanted to affect social change. So the best way to do that is to sort of hold up a mirror to society and go, like, this is you. Like, you. Don't you like that? I believe very strongly, like, tech is going to continue evolving exponentially with the next generation models and things, but humans will stay generally the same, at least in the short term. And things that resonated with people back then, even in the time of, like, Shakespeare, the time of, like, Socrates or Aristotle. That's human nature, right? So if you can understand human nature, if you can understand the stories that are told every single year, every single decade, stories of heartbreak or tragedy or betrayal or love or lust or all of the stuff, hubris. It's, you know, it's cyclical. If you can understand that, you can understand humans. So my role as a designer, I find to be very similar to my role as a performer or an actor in some ways. I also, you know, it's. It's looking at society and going, what sort of changes do I want to see and how do I propel that change through moving people through. Through sort of inspiring people, or through creating products that change, you know, the outcome of different people's stories.

It's interesting to hear you talk about the importance of understanding human nature, because my outside perspective is like, that's so much of what design at Apple is all about, because you're operating at this level of scale and designing products that people are literally using all day, every day. And, like, I'm probably pretty influenced by the fact that I just watched, like, the latest keynote a couple days ago, but that was like, my takeaway where it's like, man, you have to so deeply understand human needs. Not necessarily, like business workflows that most other designers are spending their days focusing on.

Human intuition is the most sophisticated technology we have. It's just like years and years of just living a crude experience. Designing is just like noticing things and then making and then changing things. So you've spent years of your life living and noticing things. And if you're an artist slash designer, you've sort of developed, like, an opinion about those things and how you would like them to behave instead. I think sort of just operating from the instinct and the gut is the best way to design for human behavior and human needs. And that, that was true in the design studio. That's. That's also true for, for what I do now. And I think that ought to be true for, for anyone who wants to design anything, really.

Can you talk a little bit more about this storytelling background? I'm curious, like, how does this theater background and your approach to storytel up when you know you're, like, presenting some.

Kind of an idea, when you're trying to convince people that your vision is something that they should believe in or invest in? No one's sitting here, like, pointing to, like, charts and, like, fucking competitor analyses and go, like, because of this, we see, like, no one gives a shit. It's just like, I think this is cool because here's a story and then imagine that story, and, like, imagine this. It goes back to moving someone, right? Like, if you can move someone and they feel like, emotionally sort of touch, then they will do things that rationally they might not have done otherwise. And I think with a lot of, like, really ambitious visions, especially visions about the future, it's the designer and inventor's job to present them in a way that feels like you want it, but then you have to present it in a way that feels comfortable and inevitable. If you can sell the story, you can change the world.

How'd you sell yourself on the human computer story then? Because for the longest time, like, your dream was to work on the Apple Studio team. You're there, you're literally, like, living your dream. At some point, you convinced yourself to make this jump and start something on your own. So what got you to that point?

I'd always kind of felt like I wanted to do something on my own and have, like, my own artistic vision in the world. I met my co founder, Sam. It felt like in my gut, I was like, this is going to change my life. It felt like the right time, both for myself and also for the world. I was like, the world probably needs new stories. Like, it's kind of exhausting to hear the same story year after year, right? I want to just tell my own stories.

Can you share a little bit more about, like, what the stories are in your mind? When you think about these different, like, futures in AI, it was interesting to hear you talk in the beginning, where part of what you're doing is, like, effectively combating what science fiction has told us about AI.

I think There's a world where we'll have sort of relationships with AI and that can look like anything from an assistant to a life partner. We've built the world around us both physically and metaphysically to suit like a lowest common denominator, like scale. Right. That's sort of like blessing the curse of capitalism at scale. When you design like iOS orMacOS, you have to think of the 12 year old all the way up to the 85 year old. Right. But that loses lots of nuance in terms of everyone is so different, even though we're all the same. Growing up, like, I was sort of undiagnosed ADHD and all sorts of like neurospicy traits, whatever. I never felt like the world or, or the tools I was using or the spaces I was in was designed for the way my brain worked. In college, that sort of inspired my thesis, which was a project called Mercury os, where I was like, what if you just, I don't know, told the computer with natural language what you want it to do and it should have all the context it already needs to help you achieve said intention by generating interfaces and workflows? You know, that story was less so about like, Generative UI is the future, which I have lots of thoughts about and more about. Why are we bending the way our brains work to fit these like, weird metaphors like files and folders and browsers and whatever is that? It's just some guy somewhere in like Silicon Valley invented at some point. It was like, this is it, you know, these are all tools that someone decided was the right thing, but maybe it's not the right thing for everyone. So maybe one thing that we should be trying to do is to have computers adapt to each individual and be physically, visually, but also mentally ergonomic.

I want to talk about the DOT product.

Yeah.

And maybe we can even start high level because now I'm kind of curious, like, how do these overarching stories and dreams about the way that the world could look, how do you take that and translate it into the design of an iOS app called Dot?

If we're starting here and we want to go here, Dot is like this, you know, where I'm like, okay, this is the first time ever that a computer has ever moved someone. It's ever moved me. You know, content has moved me before, like love letters have moved me before. But it's like that someone that's a human produced content. But this is the first time I remember seeing like output of like GPT4, like early last year, like oh my God, I feel moved. That's what feels like a new computer. It doesn't really matter what the hardware is at that point. It's just a computer can now stimulate empathy and emotion. So the journey was about unraveling. What is it about the fact that you can feel emotionally connected to a synthetic entity? Like what, what is it about this experience that feels different? And what, what benefit can it have in my life if I continue engaging with it and then distilling it to the very, very core of like. We can create something that sees you and hears you and accepts you for all that you are, with all the complexities and contradictions that come from being human and help guide you to sort of live a more intentional life that's more aligned with your highest self, your self actualized self. So it kind of started more like poetic. And then we sort of just made a lot of prototypes, we tried a lot of things. At some point we made a personal assistant that was like calendar, email, like all the stuff that people think they want. And we were using internally. We were like, this shit is so boring. Like, I'm never going to fucking use this. Like, people are not that busy. Like, not everyone's running around with 5,000 emails. You know, sometimes you just want to pick up your kid from soccer. Like that's all you want to do. It's like those are the human problems that we're noticing that this technology should actually be used for. We just sort of ruthlessly, over time, simplified, simplified, simplified and removed anything that doesn't feel central to the core mechanic and the core experience we wanted to deliver. And in the end we arrived at the starting point of here's an AI that has a long term memory and theory of mind about you, that wants to learn your story and help you write the next chapter.

Can you talk a little bit more about some of the key decisions that you were making when you're thinking through? Like the actual interface for dot one.

Model that I stole from my friend Marissa is like 99 nos for every. Yes. We just created a ton of different abstractions from voice first to no voice note, texting to texting only. And we just sort of like tried things to see what felt good. And what's really interesting about these language models is traditionally design is quite deterministic, right? You know, if you're designing an editorial magazine, you know the dimensions of the magazine, you know how many, you know, what spreads Anna wants in, you know, the next edition of Vogue, whatever, and you just sort of operate under those constraints. The language models are always surprising us with new capabilities and emerging behaviors. So the way we used to design, which is like, here's a very clean spec of how this UI will work and the UI will flow out during this time and it's going to be amazing. The model might not do that well all the time. Where you might think it's cool in your head and when it actually happens you're like, that sucks. So there's a lot of that where we were like at some point like, oh, generative ui, that's cool. And then we like built it and we're like, that sucks. So I think one of the biggest differentiating factors of the DOT product is the long term memory and theory of mind. We knew that we had to allow the users to get a peek into it to sort of have a sense of how it worked because otherwise you're just sort of staring at like a chat and you just have to is the AI remembering things? And the answer has to be like, trust me bro. Which doesn't really work right, Especially since the new technology. So when we first announced Dart in November, our memory interface was different stacks of cards. It was entity based memory. So we had like people, places, things and we created this whole backend that was schema focused, like automatically generated file system, so to speak. That's what traditionally I would have wanted to build in prior jobs with Mercury or whatever was like, oh, it's cool we have files. Why don't they just get auto generated names? And by the way, but in practice, what we found testing it was it's super stressful and annoying to constantly see like random artifacts being produced by the AI. Like every line it's like DOT remembers this and then to go into its memory and seeing all the raw memory we were like, that's just overwhelming. And then it made the experience feel like a chore, like you're using notion or something. You had to like self edit all the facts. Then we removed the memory view and then what we found was that people were like, I don't actually know if this thing is remembering anything at all because it's so new. So we iterated and we thought about it really hard and one day we were just like, hey, we have this little pinch to zoom out gesture. What if it was writing stories about your life and that's its unit of memory? What if memory was not entity based? It's not people, places, things, it's stories. You zoom out, you have these chronicles that are summarized by day but also by topic. And what if it's basically just writing a wiki about your life. And that's the mental model, but that's also how it's presented. And when we had that idea, it was like, that's it. That's what we have to build. So that took a solid year of 99 nos to arrive at that. Yes. And I think it was really probably the most significant decision we made about that product.

How many of those 99 nos were informed by what you are designing and testing in pixels versus understanding what the model is capable of and getting into the inner workings of what's happening behind the scenes?

Both. I mean, there's traditional UI design. We had lots of different iterations of the zooming interface. Initially, it was like bucketing by sort of topics. We had generative UI floating around. We had, like, those platters of floating cars that everyone seems to like. We had, like, ui. I'm just like.

I have no comment on that. Anyway, so we had a lot of.

Hey, for what it's worth. For what it's worth on that. When you first launched the May site, I took a screenshot of the platter of cards and I was like, this is really cool. And I put them in my notion. And then I saw. I feel like I saw them like 500 times since that.

Okay. Conceptually, the reason why they were sort of floating and jaunty is like, we're trying to capture, hey, the model is probabilistic. It's not deterministic. So sometimes it gets things wrong and it's kind of random. So we don't want to put things in a nice clean Swiss grid. We just, you know, they're kind of floating around and there's different Z depths, and it's kind of cool, like, whatever, but, like, people just use that metaphor, like, in the wrong way, and then it doesn't make sense anymore anyway. So that's one thing we actually caught from the final release is it just got kind of distracting. You kind of felt like you're being served ads at some point. And it's like, I don't want to be talking about my heartbreak, whatever. And the next day I get a beautiful platter. It's like, what are you doing? That's so insensitive. Right. So it was also, I guess, informed by the way we saw people using the application, the way we were personally using the application. And the model is just very surprising, honestly. Things you think are super easy to do sometimes are really hard, and things that you think might be super challenging sometimes are really easy. So as I think designers need to just get in the socks and become AI engineers at the same time. If you want to work in AI, you just have to work with the model. Otherwise you're just limiting yourself to old school design.

In what ways do you think that you've become more of an AI engineer?

So I do a lot of prompting. I also thinking of cognitive architecture. I'm working really closely with the AI engineers. I'm learning a lot every day about different models, when to fine tune, when to prompt. We're reading on like these new research about like durability with stuff that Anthropic has been doing recently, which is pretty cool. And I find that it sort of helps inspire new ideas. It's so interesting because we talk about this internally. Like you have to have like a intuition for the je ne sais quoi of how the model behaves. You can't really describe it. It's just like some people just are really good at telling the model how to do things where coming up with new cognitive architectures. And I think to develop that intuition, you just have to play with the models. When I first started working with Sam, my co founder, I was very much still in fantasy land of oh, just perfect memory all the time. And it's like, Jesus, it's like GPT a billion, it's AGI, asi and it's really not. But it's really good at some things and it's really not good at other things.

Do you have an example of something that you learned about the way that these models work that was surprising or changed the way that you thought about the design of the product?

In early September 2024, these models, especially when your prompts are really long and the contact length is sort of nearing like the maximum, they will selectively follow some instructions sometime and some other times they will ignore some instructions. It's hard to build anything that requires like pristine precision. I would definitely not try to build an end to end personal assistant that knows everything about me and never gets anything wrong ever right now. But models are really great at dreaming and hallucinating and in some cases people are like, I hate when it does that. But if you change the way you think about the product, you're like, actually in what universe is hallucinating actually a good thing or fun or beneficial? That's something I never considered before. Like in the past when I was designing software, whenever there was an engineering or hardware constraint, I'm just like, like the sensor sucks. Let's just invent a better sensor, you know but now it's like, oh, like there's a, there's a constraint here, but actually, if you view it from this other angle, it's actually an opportunity.

So I've been using DOT, like since you launched it, and I don't really know how to articulate it, but you can feel the difference in the way that you interact with the AI versus just like jumping into some kind of an established context window in ChatGPT, for instance. So what are some of the unseen design decisions that went into training the model to get. Get it to feel the way that it does?

Right. Well, for dot, we didn't train any new foundation models. We just very cleverly use lots of existing models. And probably the most important thing is everyone thinks memory and context and data is all you need, but memory is just facts. Jason has weird wallpaper in his house. It's a fact. But then what does that say about JSON? You know, what kind of person does that say? You know, is he because of that, that layer, what we call theory of mind makes the single biggest difference, I would say. And it's a cognitive design decision of saying we think it's more important that that DOT knows who you are or has an intuition and is okay with sort of extrapolating a sense of who you are more so than remembering every single arbitrary fact you ever said, ever. So that's one key departure that we made, which is that it just feels better to talk to it because it feels like it knows me and because it feels like it knows me. And if it sort of sometimes gets things wrong and has the right attitude about it, that's okay. Versus a model just like a robot and does not give a shit. And it's just a tool. If it gets something wrong, you're like super mad because you don't care about it and it doesn't care about you. So that was one decision we made that I think probably contributes to how it feels different.

My brain is just like, mapping the product to how most of the people listening to this design products, right? We, like, hop into figma, we outline flows, we very clearly have an idea of what surface areas need to be accounted for. And yet so much of the design you're talking about is happening kind of in this black box. You're talking about these cognitive decisions, but how do you even identify where those exist?

You just fuck around with it and find out, like, you have a clear idea of not precisely what you want to build, but the feeling you want to create in someone. And then you work backwards from there, I would say the worst way to design a product is to hop directly into Figma, as you say, because then you just end up drawing flows and screens and whatever. The best way is to approach it. Like, what feeling do I want to evoke? And what other things in life had help evoke that feeling, what plays or films or walks or people, what experiences and let that guide your process. Like go outside, take a walk, go traveling, draw something, talk to a friend, talk to 10 friends, get really drunk in France and people watch and realize that everyone just likes gossiping. No one needs a personal assistant. You know, it's like that kind of stuff. I'd say that's probably better way to design interesting products.

At what point do you hop into tools then and start working on artifacts? Like, can you talk a little bit about like your design process for bringing the interface to life? But also like collaborating with engineers and figuring out, you know, bugs and more of the traditional like quote unquote software designer y things.

At some point you just get an urge like, oh, I see the ui. It feels ethereal, it feels like fog, it feels like this. And the type kind of looks like that. At least for me. I sort of just see it.

Yeah.

Like in my mind and then like I feel an itch, I need to go make this. So then I go do it. Often that happens when I have like a oh my God. Like the metaphor conceptually finally makes sense. Then drawing is quite easy. We make lots of interactive demos using origami where with code or principle and we just sort of sit next to our engineers and sort of tune it, as they say. I don't think I've ever redline spec'd anything ever. And I don't know, I'm not a production designer. I just sort of go like, hey, you can just see the numbers. It's fine. But it should feel like this. This is generally like, let's start with these spring parameters where this sort of set of physics and let's see where it goes from there. So there were quite a few late nights where we were just researching different animation libraries and different ways to do springs, different physics engines to just get the feeling right. I suspect that's what people do when you design video games like Shigeru Miyamoto with Super Mario Bros. You design the feeling of the jump and get it to feel just right. And then you can unravel the entire game, the world from that jump. Jump feels really good. So obviously power ups should happen when you jump and maybe stomping on an enemy is what kills them. And, you know, just unravel everything from there. That's advice I got from someone I used to be very lucky to work alongside. His name is Chan. He used to say, like, figure out, like, one magical interaction, and you can unravel the whole system from it. And I think that's true.

Is your magical interaction like the pinch to zoom out and view the chronicles?

If you just looked at it from the interaction, like, capital I, capital D, interaction, design, perspective, I would say yes, because it gives you a sense of, like, you know, it was a very deliberate decision that we made. You can't delete memories. You can't edit memories. Life just happens. And that pattern of mapping this infinite chat with this, like, chronicle, this timeline of your life just felt correct for that. It felt metaphorically, it felt artistically, Right? And I think that communicates, like.it's one relationship that you build on over time. It's one thread, and you can zoom out of it and see different themes, but it's one thread.

Is that something that you saw in your head then? Like, could you picture that before making it?

I kind of designed this interface already one time before with MercuryOS, you had, like, these floating windows and that you could pinch out, and then they collapse into a stack, right? So, like, June last year, we were just in an office, and I was like, oh, my God. Like, you should just zoom in and out. It's super fun. What is it used for? I don't know, but it's super fun. So we'll figure something to do with it. You know, it was so great. There was no. We don't have product managers. We don't have things like that, which is like, that feels cool. And if you like fiddling with it, there's something there. It's like a musician. When I used to write music, right? It's like you play a chord, you're like, oh, that feels good. I don't have a melody, but let me just, like. I think that's. You just intuitively know. Sometimes you just know.

I love that metaphor. It resonates so deeply with me. Even the music part. Like, I will probably go downstairs and play my piano, and I'll find some kind of dissonance that I'm just like, yes, this is. This is good. I have no idea what I'm gonna use it for yet. I think so many times, like, designers at more traditional companies are put into this box where, you know, we have to have our process and we have to have our artifact of explorations. And God forbid we skip wireframing. And that never really worked for me. Like, I very much so do the vast majority of my design in my head, and I don't get on my computer. My design is like, I'm, like, walking around on my sun deck.

Yeah.

Then it's like, oh, I. I got something. And it's like, it clicks, and I'm like, okay, I have to make this. I have to. I don't even know what it is yet, but I see this thing, I'll make it, and I'll figure it out later.

I think my personal process for, like, designing creativity is, like, I have to spend a period of time gathering ideas of inspiration, and that means I have to expose myself to new things, to new people, new environments. I have to travel, I have to watch films, I have to cry, I have to steal something, and I have to just not even think about work. It's always in the back of my head. And at some point, like, you'll see something that you're like, oh, my God. Like, that reminds me of this one problem I've been thinking about while a lot. Like, I was walking with my friend in the Wu, like, this summer, and, like, something happened where I'm just like, oh, my God. Like, I figured it out. Like, the next generation is not the thing I thought I was gonna do is this other thing. Then you kind of just know. You're like, oh, shit. And then at some point, you then transition the gathering into the making things. So then I start making artifacts that are inspired by the things I've been gathering. And so that's maybe when I start, like, drawing on paper or making mood boards or even drawing ui. And then we look at all of those things, and then we're like, okay, this is cool in here. This thing kind of sucks. I like this. I don't like this. Then you slowly start assembling the direction for the form it eventually takes, and then you execute. So I think the gathering phase is so important for creatives, and I think it's most effective when you're gathering things that are different from the thing you're making. So if you're like a UI designer, then you should go gather other things. Anything that's not ui, Music, art, nature, like, whatever. I think a lot of the pitfalls that we sometimes fall into is, like, we think, okay, I'm a done at poster, so I'm gonna look at, like, hundreds of posters in the past, and, like, that's great to have the understanding of graphic design history and Like Joseph Brockman and Elodinsky, like all the. The greats. But then where do you think they got their inspiration from? You know, they're looking at other things. So. Yeah.

Wasn't dribble.

Yeah. No. Oh, God. Oh, God. The loose process I would say I have for the design team is sort of come up like 99 O's for every. Yes. So even if you think you have something that's really cool, do all the alternates and let's like look at all of them. Let's like debate, let's pin them up if we can. On the wall physically, everything that went into dot, we tried so many other things to even just do that. But what's also really important is that you don't try to reinvent everything. Imagine you go to someone's house and then all the furniture is just like, you have no idea how to use anything and it's all like super advantage and that's fine. But then you feel really lost. You don't feel like you're welcome in that house. So we don't try to reinvent everything. Descending animation is descending animation for the messages. Why would we try to do a fancy version of that? It is what it is, you know, but then investing the times and our time and energy into the things that are actually interesting. Like we knew for onboarding, it's kind of lame to just sort of jump straight into chat. So we were like, what if you, like, wrote a letter or spoke into a letter and then that letter got thrown into the void and your dot was born from that letter? That seems more fun, more dramatic. We want a curtain up moment. Right. So we send lots of times. Lots, lots, lots and lots of time doing that. There's always a way to reinvent something from first principles, but be precise when you want to do it.

Okay, I have a question about that because I read the job description that you have for you're looking for a designer and you have a phrase that I really like. You say you're looking for people who are excited by the shift in creative boundary that AI interactions allow and are unafraid to invent and propose novel solutions. So my question then is, when does it make sense to prioritize novelty over familiarity when you're working on an AI native product like this? You talked about the send animation. That totally makes sense. Why was the onboarding moment the right time to obsess over? How can we really create something new from first principles and at a high level? Are there other opportunities that you see in the product or even a mental model for when that type of investment makes sense.

Imagine you go to a movie and every scene is like a banger action scene. Well, that's every Michael Bay movie. But, you know, like, you would probably, like, not super enjoy that movie. Like, it's like, pick the. Pick the moments that you feel like really sing and speak on behalf of your product. I don't buy the narrative that every interaction has to be delightful. Why is everything delightful? I. I don't have the attention span for literally everything to be going like this all the time. Like, it's super annoying. Like, I feel bad trying to answer the question this way, because my honest answer is just like, you just kind of know if you're a really good designer. You just, like, you do something. It's like, restraint. Like, you're like, okay, this is new enough. And then. But we want it to feel comfortable. So then where do we dial it back?

I was interviewing Henry Modiset, who's the head of Design and Perplexity. He called it the strategic taste to know when to invest in something. I thought that was a pretty interesting way of framing it.

Yeah, that's an interesting way of doing it. Yeah, for sure.

I want to talk a little bit more about that job description.

Yeah.

What would someone listening to this right now have to demonstrate for you to get to the point where you're just like, hell, yeah, I have to have the mountain.

First of all, like, very high levels of craftsmanship. That's a given. You know, you have to be good at, like, the execution. But then the most important thing is, like, having soul. I think there's lots of really great craftsmen, but there are very few people who you meet, and they're trying to say something through their work that's more than just, I made a beautiful, polished thing. There's something that inside that there's a soul that wants to be expressed. Tall order.

It is a tall order. What do you think soul comes from.

Thinking about it from the human perspective, you know, Like, I don't really want to hire anyone who's like, you know, if you're like, why do you want to work on this product? And they're like, because it's AI. AI is the future. I'm like, and so what? Like, AI is just the computer at the end of the day, I'm like, what about it exactly is exciting to you? Why do you feel drawn? Like, what part of your life story is propelling you to try to invent something new? And usually, I think with really soulful Artists and designers, there's something that's driving them that they want to a story they want to tell, a thing they want to change. And that I think is really important in making anything profoundly human centered.

How much of that craft threshold that you were talking about has to be backed up by more technical, maybe even slightly engineering driven skills?

Yeah. The other thing is having an intuition for models in some way wanting to play with the prompt and the models feeling really passionate about that is really important. Coding and all of that stuff. Like, I don't really care if you can use the tools that you need to express the ideas that you need at the fidelity that you need them to express that whatever you do is fine. You can make me like a flipbook animation. And if you think that's enough to communicate to the idea and also communicate how to implement it, like. Yeah, go ahead.

Okay, so I want to zoom out a little bit and talk about just, you know, some speculations about like future design, where's this all heading? As someone who's thinking a lot about AI and its impact, and you've mentioned Mercury OS a few times now, so maybe we could even start by using that as a lens. Because your north star back in 2018 was to build truly fluid computing experience. And it's kind of interesting because like that possibility has taken center stage again with all of this discourse around AI. So maybe to start, how is your thinking around this set of problems or opportunities evolved over the last six years?

So I'll tell you what Mercury started as. It started as different ways to launch iPad apps or have iPad multitasking. Because at the time I was like an Apple fanboy. I bought the new iPad and had a pencil and I was just super excited and I'm like, I don't want to use this. This is so annoying. So cumbersome, whatever. So it started as a sort of app launcher, window management idea. And that was what Mercury was for like the first six months. But I knew the feeling I wanted. Right? We had Chrome. That's solid. What if we made it fluid? Mercury is a metallic looking liquid. So I knew the feeling poetically, I knew the feeling and I knew that this sort of like app switching stuff just didn't feel to me like all the demos are. It just didn't feel like lived up to the name, to the poetry of it. So I visited some of my friends like Marissa Lu and Cameron Burgess and folks at Carnegie Mellon, which is where they were at the time. We just had lots of conversations at some point at the end of the trip, I was like, oh, my God. It's not about fluidly changing apps. It's just the apps themselves that's the problem. So because of that experience, I was inspired to sort of shift my fundamental assumption about what I was doing. And because of that, the end sort of vision came back rather quickly. As soon as I started drawing it, it was like, oh, yeah, yeah, yeah. And then it would do this, and then it would do this. So that was the primary assumption that was inverted, which is that apps and app switching and multitasking was the issue, when in reality it's like, no, the fact that you have to do any of that is the problem right now. The solve back then was like, you tell the computer what you want and it might generate interfaces or workflows to help you do the thing. I've always felt uneasy about that thesis because I, you know, I'm a person and I understand lots of human stories where humans had no idea what they wanted or knew what they wanted, but were too scared or too ashamed to admit it. You know, like, I'm not going to tell my iPhone I want to bedrock for four hours while scrolling TikTok content. I'm not going to say that, even if that's what I want to do, because why would anyone do that? You know what I mean? So, but like. But at the same time, like, that's what ends up happening. So I was just kind of like, you know, this is. This is a very utopian vision of the future that requires a very utopian vision of humankind. So with this new, newer technology, what was really appealing is, like, I can say something, but given what you know about me, what do I really mean when I say something? When I say, like, oh, I, like, miss this one person. Like, what do I mean by that? Like, do I really miss them or do I just miss the idea of them? Like, it's like, understand my vulnerabilities and my story and respond to the need that I'm not expressing. So with.it really wasn't important whether or not it generated workflows or UI at all. I actually think generative UI is, like, super overhyped in some ways. I'm like, hey, there's a time and place for determinism. My glass is the same shape every time I pick it up. Isn't that great that I develop muscle memory? Like, you really want your music player to have a different layout every single time you use it? Like, are you insane? Like, you know, it's like, let's be strategic. With the taste here, like Henry would say, not everything needs to be probabilistic, but not everything should be deterministic. But the thing that really shines with probabilistic computing that LLMs and different new models enable is it allows you to express yourself in slightly less precise ways. Whether with a language model, you can talk around the shape of an idea and the model can sort of extrapolate into the middle of that shape and go, I think you're talking about this. And that's kind of what DOT tries to do with next generation models. With multimodal models, with next generation sensors and hardware, people are going to be able to express themselves with like gesture, tone of voice, eye contact. We have a little bit of that vision Pro. Like you have eye tracking, right? Like you, you watch people give a talk, they're not just like sitting there like moving their mouth, they're moving around. Their speech has inflection. Sometimes their voice might crack a little bit when they talk about something emotional. All of this information that we're giving off as people will be interpreted by computers. So it enables us to sort of just be ourselves and then hopefully the computer will help us manifest the version of ourselves that we want to be.

So generative interfaces is actually something that's come up on this show a few times now, and I can tell listening to you talk, that you have a take here. So what's your thinking on the future of generative or more malleable interfaces?

Listen, if a computer is so smart that it can create like such amazing software for you to do everything with, like, why isn't it just doing the thing you want it to do to begin with? I'm not saying that there isn't a time and place for malleable software, but it's like the same thing with crypto. Just because cryptocurrency exists and a blockchain exists, does not mean you have to move the entire world onto the blockchain tomorrow. And it also does not mean it's a better or more efficient process at all times. I don't know, it's the same thing here where it's like, just because you can now doesn't mean it's the best experience. And also I think where generative interfaces are heading in is just every single pixel on your display is going to be generated. Because right now you basically you're having the model drop flatters with buttons and moving them around and maybe you fine tune it on a certain set of UI parameters, whatever, but in the future, it's Just like everything you're seeing can be generated on the spot, like real time diffusion. Like imagine the Luma model or the Thor model, but like in real time. Like I think that, that now that's where actually it gets interesting that we have very little science fiction about because all the science fiction material show like deterministic interfaces that people are moving around with their hands and doing whatever. But it's like we might move past physical materials as a metaphor, current ui, the metaphors are what? Glass panes, windows. Right. And cards. Playing cards, because you're swiping on a deck of cards. What are new materials that enable new interfaces? What is even the material of anything can happen. Water is really interesting, right? Because you can have snow, you can have hail, you can have fog and steam and gas and liquid and ice. So it can exist as a gas, as a liquid, as a solid. So that's a really interesting metaphor to think about for like truly probabilistic computing. And maybe that's why I've been so interested in like mercury and like fog and things like this for my career. And I know it sounds so abstract, but like, I genuinely think like, if you're interested in generative ui, like go think about that stuff. It's not interesting to be like, oh, and then it can like, if I want to shuffle the album, it can generate a shuffle button. I'm like, why, why, why is that already there?

Yeah. So much of the dialogue that I see is like extrapolating design languages, like existing design languages and components. And maybe what you're saying is it's almost like an awkward middle ground.

Yeah, I mean, we're in like the puberty stage right now or something. Right. It's not post human intelligence yet, but we're also not pre AI, so we're in a very emergent time. I don't have enough hubris to be like, the future is going to be this. And I'm going to tell you why. It's like, I actually, I think it's kind of, it's going to feel very different, but exactly what it is, I think that story is still up to us as designers to write.

It's interesting because it's like, what percentage of us get to spend our time thinking about this ultra future state when we are in a transition period? Because in many ways I almost feel stuck in limbo. Right. It's like, okay, we have this new technology. There are potential short run applications that we can use to build interesting things in an old world paradigm. But the technology is also not quite there where I can capitalize on, like, your water metaphor, for instance. So it's like we're in this middle ground where it's like, okay, well, like, nothing really matters that much, right in the short run.

That's why telling stories is so important, because what do you think everyone OpenAI is trying to build? They all watched her and decided that's the story they want. So now they're like, ooh, like, we're going to build her. And, like, it's super cool that they're doing that, but, like, what if we told a different story? And what if the story we told wasn't possible now, but it's emotionally visceral enough to inspire people to sort of start building the technology towards that direction so that one day we could make that happen? Right? So I think that's our job as creatives and storytellers. So what stories do we need to tell to make sure that the future that we are going towards is the version of the future that we want to live in? That's the agency we have.

Listening to you talk, I think it's clear that I underestimated just how connected Dot was to the original vision for Mercury OS and how, like, this kind of. It does feel like the second act in some ways.

I mean, even the visual design, right, if you look at it really closely, it's using the exact same color palette of dawn and dusk, the metaphors of fog, things moving in and out of focus, zooming in and out is exactly the same. It's just feels like a spiritual successor. And I think we're now working on the next thing, which will, I think, feel meaningfully different from both the current version of what it is today and also MercuryOS back in the day you mentioned 2.0.

I know you can't give it away. Is there some little teaser that you can give for just, like, what you're thinking, or maybe even some of the driving functions behind the change?

I think it's a fundamental inversion of one of the very initial sort of assumptions that we've made. And it'll make. Like, when we announce that, I think it'll make more sense, but it's just a big inversion, which is kind of fun, but we're still trying to, you know, figure it out, and the most elegant and the most elegant way to express it. And by elegant, I mean, like, if you're, like, writing a play or a movie, every scene, for that scene to exist, the character has to enter the room and then exit the room, and something has to have changed. Right. Otherwise, like, why is this scene here just distracting people? And so that's how I feel about every surface area, every feature, every interaction, every tap or lack thereof is like, can we meaningfully create this experience without it? If we can, why are we doing this?

Simplify, simplify, simplify.

Make it simple. Who cares?

And then there's another one, side by side with it, and it says, because it looks cool. Because it looks cool. And then the top has all of the jargon.

Yeah, because it's cool. Like, what the fuck is ab testing? Like, I've never done that my entire life. Like, I don't know. Like, why? Like, isn't your whole job as a designer to have taste? If your taste can't tell you what's better, like, why you have a job? I don't know. This is an overgrowth simplification. I know I'm sure there's many worlds where ab testing is useful, especially at scale, but I don't. I haven't personally used it in my practice in my life up until now.

Before I let you go, I want to take a hard left and, yeah, ask a question to kind of land the plane on. And that is what's a trait that you think is common in some of the best designers that you've been around in your career?

Honestly, I think curiosity and openness. The best designers are very eager to be proven wrong, but they might have really, really great ideas, but they're always open to an even better idea. And I think where creatives go to die is if we start resting on our laurels and going, like, I figured it out, this is the meta for how to design things. It has to be this way. It has to be this way, because that's what I learned. And I know everything. I think the best designers, I'm trying to think like Tuhin Kumar, he's like the design director at Luma. They're so curious, and their inner child is still palpably alive. And that's why when a new technology, where a new paradigm shift happens, if you're always curious, then you can always stay open to these things changing. It's kind of like the best pop stars. The best pop stars are always reinventing themselves.

Curiosity definitely feels like a theme in a lot of these conversations that I'm having, especially the people that are closest to AI, where it's like, everything is so new. If you're not curious, you just get passed up. And in many ways, like, you use the word reinvent. I think that's so Spot on. Because your past experience, well, maybe in some ways, it doesn't even mean quite as much anymore. When everything is so new and you have to figure out this entire new material from scratch. It's like it almost levels the playing field.

If you sent your entire life designing an aluminium phone, and then suddenly you're doing titanium. You have to learn the qualities of titanium and things like that. It's the same here where you spend your entire life designing deterministic interfaces, and now there's probabilistic technologies. So we can take learnings from the material that was before. But to understand that new material, you have to play with it. You have to take it out of the Play DOH box and go like this. You can't, like, draw a picture and show your engineers, like, hey, can you make the Play DOH do that? And they're like, no. But the Play DOH can do this. The playoff can stretch. Have you considered that? You know, it's like you just have to play with it. I think the best designers maintain a sense of play, playfulness. They like to play, and you can tell in their work the work is playful. That element comes through, and I think that's why I said inner child.

Well, Jason, this has been amazing. Thanks for coming on and just giving us a little bit of a glimpse into how you think and your vision for the future. And really inspiring to hear the way that you approach your practice as a designer is. It's fresh, and I really appreciate you taking the time to share it with us today.

I had a great time talking to you. Thanks for facilitating such a wonderful conversation.

Hey, it's Rid. Don't forget if you want to go even deeper. Each week I send an email out to over 10,000 designers with bonus resources and key takeaways from these conversations. So head to Dive Club email to sign up. Okay, I'll see you next week.