Design for Augmentation (not Automation) | AISC

Awesome. Hello everyone, thanks for joining us today. I'm very excited for this session with Amin. Believe it or not, I've been trying to trap him, drag him into giving a talk for what, like probably a year now. So I'm very excited to have you with us today, Amin, and very excited about the topic that you're going to speak about. I think I came across your name when I was sort of starting to learn product stuff and people mentioned your name as somebody to go to to learn about all topics product. And you have a phenomenal background. You have currently you're a co-founder and CPO Chief Product Officer at Converse, which is a few months old company. I hope that you will tell us more about that. And prior to that, you've been, you know, in various product lead and vice president of product kind of roles in previous companies. And I know that you're going to talk about that a bit, so I'm not going to go into details of that. But, you know, I really also like that you teach Product at Product School as a principal lecturer there. So I'm very excited to hear the topic today and I pass it on to you. Awesome.

Thank you so much, Henry. So kind of you. And it means a lot. A lot of things that you said, it means a lot coming from you. Yeah, it's a topic that I'm really passionate about and something that I've talked to a lot of people who have been in the space. And last year, last November, when there was a thing for in-person conferences, I also talked to a lot of people. Product Management Festival after I presented this topic. And a lot of things I'm presenting today are things that kind of iterated on that presentation and that talk that I had. And I hope it's going to be educational and hopefully people get value from things that we learned along the way at different companies that I was at and they can apply it to what they do. So as you mentioned, I worked at a variety of different companies from B2B to B2C to B2 C-type of products. And I'm going to use a lot of these case studies that I'm talking about, and the learnings that I'm talking about are things that we learned, sometimes learned the hard way, through building AI-powered products. And I'm hoping by sharing those, everyone can use these learnings to be able to build better AI-powered products and be able to apply some of these learnings to what they do and what they build. And a lot of things that I'm mentioning today are kind of summaries and things that I've covered in a book that I recently published on Design for Augmentation, Not Automation. And this is something that again, really kind of all the conversations I had at the Product Management Festival got me really to kind of write this and share some of these thinking around why it's so important to think about augmentation when it comes to building products that are powered by artificial intelligence. And the COVID-19 pandemic and what we're going through right now is making this topic more and more relevant. Right now, the world and a lot of these corporations that are out there, they're all thinking about how can we— sorry, let me just disable this— they're thinking about how can we go about automating what we do and use artificial intelligence to essentially remove a lot of the humans in the process and make sure that we can maximize our automation. And if we don't have product leaders that think about the value of augmentation and how to focus on augmentation instead of automation, we'll end up with products that do not add as much value as they would if the product leaders that are designing and building these with their data science teams thought about augmentation. And right now we are seeing a broader shift towards this automation, and the economy is under a lot of pressure to focus on this and making sure that the economy is focusing on maximizing automation. So again, a lot of things we're talking about are relevant to today's world and what's happening today and what we're going through. And the 5 takeaways that I'm hoping that we can have by the end of this talk is the first one is around we need product and design leaders that have a strong product sense, and we'll talk about what product sense means and why that's important to harness the power of AI. And we'll talk about why it's important to fall in love with the problem, not the AI models that we build, and thinking about how we can use AI to augment the product experience for our users the next time we are designing a new feature. And talking about what the role and the importance of product leaders' responsibility for handling low performance scenarios or failures when our AI models do not perform up to expectations, how we go about handling those low performance scenarios. Scenarios. And last but not least, learning about how we can leverage our product sense as product leaders to use AI to augment human intelligence. So product sense is something that a lot of companies right now in the world are focusing on. So Facebook, this Facebook product management interview process is primarily, one of the things that they primarily focus on is product sense. Product sense is the ability to consistently build for value. It's one of those skills that all of us can develop. It's not something that, again, you need any specific type of education or any specific type of learning to go through to get it. It's more about having hands-on experience with building functionalities and building features that end users and people end up using and provide you feedback. And as you go through that process of building, measuring, and learning, and building more functionalities and more features for your users, you get to a point that you learn about what's working and what's not working and what are the type of functionalities and AI models that you can build that do lead to value creation. And as product leaders, all of us, we rely on our product sense every day to make the right decision for our users. And we always, always focus on the users and we know everything else will follow. And this is something that is really important for all of these companies that are focusing on automation and leveraging AI for their day-to-day activities to think about where they can use AI to add value to their ecosystem and to their product offerings. And what it really entails is creativity, domain knowledge, intuition, reasoning, and most importantly, empathy. One of the things that really differentiates great product companies and great startups that have AI-powered products from the average companies is that there is a lack of empathy for what they build for the end users. So when we talk about user augmentation and designing for augmentation, not automation, a lot of that comes down to developing empathy for the end users and understanding their jobs to be done, understanding what are the things that they're trying to accomplish, what are the things that they get value from, and using AI to help them to be able to do that more effectively. And the one type of functionality and type of value that can be created here is around, for example, coaching. So we can build better AI models that can act as a coaching agent for our users. So instead of completely replacing someone's decision-making and making the decisions for them, we can use our product sense to understand how we can coach our users, our end users, to help them to become better decision makers. We'll talk about it more, but that's an example of what the value of product sense, when we use product sense in our day-to-day thinking and decision-making process. And the one thing that is the connection that we've come to learn as we've built more and more AI products is that Product sense plays a massive role in building successful AI products. And at the speed that AI is moving right now, there are a class of problems, a new class of problems that are coming to action, and that's the human problem that it doesn't look like, it doesn't seem like that AI can solve yet. And the human problem are the problems that are a lot of times cannot be fully predicted and can be fully embraced with a set of algorithms. And it really requires that investigation and type of problem solving that humans have to be able to solve that problem. So one second, let me just disable this. It's a little bit distracting. All right, and the one thing that I always think about is this by Josh, who's a UX designer at Google, and he says that if you're not aligned with a human need, we're just going to build a very powerful system to address a very small or perhaps non-existent problem. And that's happening a lot today with a lot of startups, a lot of companies that are building AI-powered products. Are building products that they think are creating value, or they think that they're helping the end users to get more value from their offering. But at the end of the day, they're not really solving a meaningful problem, and they're not even solving that problem in a way that the users are willing to pay for their offerings. So at the end of the day, what it comes down to is being fully aligned with, uh, with a human need and, and building systems that truly create value for the end users. And we'll talk about that in a very, very, very specific example, so you'll see what I mean in a second. And with that, it comes down to focusing on how we can turn, uh, the AI research that we've done, our data science teams have done, into products customers want to buy, can get value from, and can rely on. And that's really the role of product management, uh, in, in AI. It's about working with data science teams from the beginning, from the discovery stage of understanding the problem, identifying the problem that we need to solve, and all the way down to what we can do with them, how we can work with them to get value, to create value for end users. The one thing that's really important to do that and make sure that's done effectively is focusing on again, the problem, falling in love with their problem. And what's really, again, differentiating successful startups and successful AI product, AI companies from the ones that are not doing so well is that lack of problem identification and focusing on the wrong problem. And the question we should ask as product leaders all the time when we're building a new functionality for our customers and for our users is that What problem is my product facing and would it be a good problem for machine learning? But unfortunately, a lot of times at a lot of companies, the question is asked the other way around. We ask, you know, how can we use machine learning to solve this problem? And the reality is that not every problem is an AI or ML problem. A good example of that is Gmail. So think of the last time you were sending an email to a colleague and you meant to attach a file to your email. And Gmail reminds you and asks you, are you sure you wanna send this email? It seems like you wanted to, you forgot to attach a file. Now a rule-based algorithm works perfectly fine here using artificial intelligence or, and building a machine learning model that can detect all the different variations and possible ways to predict the likelihood of someone attaching a file to their email is an overkill. It's a very attractive and cool machine learning problem to solve, but it doesn't create any value for the end user, and it's definitely overkill in terms of value creation and value capture. So that's an example of how we can start with the problem, ask ourselves what the problem is, and then once we know what the problem is and have a solution in mind, we can ask ourselves, Does machine learning or does AI augment this experience or augment the solution if they are being utilized? A lot of times that's not the case. You can save a lot of time and resources by not allocating your resources to solving problems like this. The primary question that comes down to is something that again, it's not groundbreaking and something that I'm sure a lot of you have already heard about or you've been exposed to, it's the idea of machine learning and design thinking. And design thinking is really the primary, the main way and the most effective way to shift our thinking from technology first to people first. And even though there's a ton of content and a lot of different resources out there that talk about design thinking and machine learning and how we can do that, always the problem comes down to the fact that design thinking is, becomes extremely hard when it gets to the prototyping stage. So design thinking, for those of you who are not familiar with the concept, is the concept of focusing on your users and focusing on a problem that you're solving and going about validating these type of problems that you're solving for your end users. And by using design thinking in machine learning, we can ensure that we are building meaningful solutions and customer experiences that delight our end users and allows us to build better products. But right now, the challenges that we're having is that a lot of product teams, as I mentioned, they're jumping right into product strategies that start with machine learning as a solution, and they skip over focusing on meaningful problems to solve. And the one thing that again, it's really easy to say, but really hard to execute on is this idea of falling in love with the problem, not the solution that we have at hand. And this happens to everyone. It happens to designers, it happens to product managers, it happens to engineers, and it happens to data scientists. A lot of times we do fall in love with the solutions that we have and this model that we've created that, when it comes to precision and recall or the accuracy, we're getting great results with, but at the end of the day, when we go back to the first principles and thinking about the problem at hand, it's not really, it's not creating a substantial amount of value for end users and it's not something that can help our business to succeed. With design thinking, like you mentioned, the one thing that a lot of people have talked about, a lot of resources out there that talk about this is use design thinking to improve your machine learning process or build better AI-powered products. But the one step that a lot of data science teams, a lot of product teams get stuck with and they don't really manage to be able to fully utilize design thinking for is this prototyping stage. The reason for that is that sometimes it's really hard to build a machine learning model that handles 20% of the cases but fails the other 80% of the time, or building an autonomous car that doesn't kill anyone 90% of the time, but 10% of the time it might accidentally hit someone. So it becomes tricky for a lot of these AI cases and machine learning scenarios to build to build a machine, to build a prototype, to be able to validate our solutions and what we are building. But the good news is that this is where leveraging existing machine learning models and fake doing and Wizard of Oz testing can change everything for your product and your company. So I'll give you an example of how we did this at CrowdDrift, but for existing ML models, For example, what we did at CrowdRift, this was based on utilizing Google's existing vision API to tag all the different objects in a photo and using their vision video API to have all the components in a video. I'll show you more examples of that and what we did, but we didn't have a data science team as we were building those functionalities. And with fake-dooring, it's about creating a fake door and making it as if it's an actual offering or an actual product that you have and just measure how many people are interested in that offering, right? So for example, if you were thinking of adding automated chat support, very simple example, to your users, that can initially be done as just a fake door. Maybe we'll click on it You tell them that it's not available in their area, like that's not, you can say that we're not offering this in Toronto or in New York City, but this will be available soon, enter your email address and we'll let you know when this functionality is available. So that's a way of measuring interest in what you're building before you go and spend time building a machine learning model to do what you're thinking your users are gonna get value from, but when it actually comes to them using the product, they end up not really doing that. And Wizard of Oz testing is about getting actual humans to do what you want your AI model to do and measuring the performance and measuring the engagement with that offering. So instead of you actually having a fully built machine learning model to do what you want your product to offer, you have a real human behind a computer acting as if this product that you have is doing that fully in full automation, but in reality you have someone doing all those tasks. And again, you can measure the performance, measure the engagement, and based on that, decide how to iterate on it and how to actually build your end model. And this approach helped us to be able to achieve product market fit at Crowdrift. So back in, uh, 2014, um, and between 2014 and 2017, uh, Crowdrift was a social media aggregator. We had a ton of, a ton of competitors in the space. We had about 16 players who were targeting the same market. We had competitors in San Francisco that they had raised $9 million, uh, seed round. We had competitors in Australia that they had raised $20 million Series A, and we had a set of big competitors going after the same market as we did, and there was nothing that was differentiating our offerings. Everyone, all of these competitors including CrowdRoof, we were all just social media aggregators. We were aggregating content from Instagram, Google, YouTube, Facebook, and we would show that to, for example, Destination Canada or Travel San Francisco. San Francisco. So someone, a marketer that travels to San Francisco, could come to these platforms and see the photos of all the travelers who've been to San Francisco and have posted a photo of themselves in San Francisco on Instagram, Facebook, or Google. And they could use that content and use that user-generated content to post that on their own social channels to promote San Francisco. And what we did at CrowdRift we, at that point, we did not have a data science team. We just had a small product and engineering and design team. And what we found out and what we wanted to validate was that we wanted to see if augmenting the marketer's ability to find the right photo, the right visual, and post, and posting them at scale would help them to be able to become better marketers. So we found out that a lot of our competitors at that point that had tried automation, had tried AI. They were using, they had built machine learning models that would look at a set of patterns for the type of photos that you tend to like or share on Instagram or different social channels. And they would automatically do that for you as a marketer. And none of them were being successful because at the end of the day, they were automating something that machines are not good at, creativity and branding. It's really hard to really have that kind of eye for visuals and finding the right photo at the right time for the right campaign to post it on your social channels. And you need that kind of human intervention to be able to make sure you're posting the right photo and putting it in front of millions of people on social channels. And what we found was that we talked to a lot of people, right? That's where the product sense and product discovery came in. We talked to a lot of marketers at Travel of San Francisco, Destination Canada, and all these different destinations that we were working with to understand how could we augment their experience and augment their marketing capabilities to build better, build a better product. And what it came down to, what they were really frustrated by and they wanted something to help them to be more successful was the ability to be able to find these photos that are being posted on different social channels and be able to post them and share them on their social accounts and be able to do that by being able to search for the photos they're looking for by using a tagging system and whatnot, which I'll show you in a second, and have those kind of capabilities. So we listened to them, we listened to them, we used our product sense, we used our product discovery process, and before even hiring a data science team, we started experimenting with some of these things that we were hearing from our users, and we started using Google Cloud AI to build and transform our product into an AI-powered visual marketing platform. And by doing that, in about a year, less than a year, we became the number one visual marketing platform. For destination marketing organizations, DMOs, all around the world. We went from about 92 destinations in 3 countries to about, at that point, was about 700 destinations in 23 different countries. And we raised our Series A, and at that point, they started hiring a data science team and started investing heavily into data science and building our own AI models. And as we did that, uh, 720 demos in 38 countries, uh, and as we did that, uh, we, uh, found out that, uh, the one thing that was really important to these marketers was the idea of finding the right visual at any time. Right now, today, more than 95 million photos and videos are being uploaded to Instagram. Just an example, just one social account and one social media to think about. Now when you travel to San Francisco, you'll visit New York City or destination Canada, you're getting millions and millions of photos posted by tourists and by locals that are posting these photos and are displaying and showcasing what your destination has to offer. And a human cannot possibly find all these right and great photos and videos at any time. You need artificial intelligence to augment your capabilities and help you to do that more effectively. And that's what we did. What we did, we used initially Google's machine learning models to find not safe for work and inappropriate content to do our tagging to make sure that If you're running a campaign for promoting Christmas in Canada, you can find all the photos that have the right set of objects in them related to Christmas. So you can find them all at once and be able to find the right visual you're looking for. And what we learned as we did that then was that we can, instead of you as a person going to Instagram and seeing all these random photos when you search for Canada, we can use AI to help you to find the right photo every single time and be able to leverage the tagging system to be able to make sure that you're posting the right photo every time you're running your campaign. And based on that, then we started focusing on image recognition. Recognition. So we realized that, you know, we can use image recognition to tag all these photos and videos that are being uploaded on Instagram. We can use smart curation to remove all the inappropriate and not great content from your CrowdRift account. And then we can also use intelligent search to help you to be able to find the right photo every single time. And as we did that, again, we used the object detection with Vision AI. We used their machine learning model and We started using that tagging system to create a new functionality for similar photos to help you to find similar photos faster. And then we did content-based recommendation. And then based on that, with content-based recommendation, what we learned was that that was one key area that we started realizing that we would need to hire our own data science team to build a more personalized experience for our customers. So if Amir takes a photo of himself in front of CN Tower and he's wearing a Nike shirt, that might be great, but CN Tower is not visible in the background. That might be a great photo for Nike, but not a great photo for City of Toronto or for CN Tower to post that on their social channels. And that's something that, again, Google had a limitation in terms of detecting the right photos and recommending the right photos. So we needed to build our own models based on monuments and based on what represents a great travel photo to come up with that kind of content-based recommendation system. And then that also allowed us to be able to find trends. So as these photos were being posted, we could tell City of Toronto, for example, that there is an event or there is a gathering happening in downtown Toronto that you can promote to your visitors. And with that, what the learning there was that, you know, we could use our product leadership and product designers and product managers to empower our data science teams that joined the team to understand user behaviors and patterns to create those highly personalized products and services, and product and services that do create value and are valuable to the end users. And one of the other things that really differentiated us was that when we started building our platform, we really thought about building an AI AI-centered product rather than AI-added product. So nowadays a lot of companies are just adding AI features to the existing product because again, a lot of investors are interested in investing in AI-powered companies, but in reality they're not even creating any value with this AI features or AI functionalities. At CrowdTruth, what we learned and was crucial to our success was building AI-centered products and reimagining how our platform should work and how it should function. And we took AI as a starting point of our design process. The same way when you're building a mobile-first product, you start with mobile, you expand to desktop. With AI-first products, it's always about starting with AI-centered product and then thinking how it's gonna work and expand from there rather than trying to just add random AI functionalities to your existing product just to make it cooler or more interesting. And a really easy way to distinguish the AI-added products from AI-centered products is just asking yourself, if the AI was to break down, would the entire product be broken? And what we learned was, as we were going through this process, is that by building AI systems with users in mind we can open up entire new areas of development and opportunities for building people-centric AI systems. What we did at Crowdrift was just an example of the massive potential in the space for disrupting existing industries only and only if we build people-centric AI systems and think a user product sense to understand how we can augment someone's experience and augment someone's ability and skills rather than trying to replace replace it in a way that it might not even produce great results. And a good example of that, again, other than the one that I mentioned, is a more simple example. So a model that predicts the likelihood of a user clicking certain videos is a really great model, is a useful action, is a useful model to build. But a model that predicts the probability that someone would click thumbs down for a specific video, it's a really interesting model to build, but when you use your product sense, you realize that it's not creating value and it's not augmenting someone's experience or augmenting someone's ability to get more value from what they do. And two other examples are that again use great product sense in what they build is Google's Smart Reply functionality. So they recognize that users spend a lot of time replying to emails and messages, and a product that can predict likely responses can save users a lot of time. Again, a really great use of product sense, thinking where the value can be created, where they can augment someone's ability and then building on top of that. And same with Google Photos, very similar to what we did at CrowdRift. The business problem is to find a specific photo by keyword search without manual tagging, which is very labor intensive. We can use AI to augment that experience. Really quick, last two slides, just lessons learned that again, I really hope that are going to be things that we learned the hard way, some, some of them, and I hope that by using them you will be able to build better products and be able to utilize them. Uh, the first one is that don't expect machine learning to figure out what problems to solve. Again, uh, we as product leaders and data science leaders need to fall in love with the problems and, uh, how we can create value for the business and start with, and start with that as our starting point. Not every problem is an AI problem. And we need to go back to our first principles, understand what are the opportunities, how we can augment our users' experience, and start there. The other thing that we learned as we were building a lot of these functionalities is that it's garbage in, garbage out. The quality of our machine learning models is as good as the quality of the data we are collecting. So it's really important for us as as product leaders when you're building your machine learning models and when you're transforming your product into an AI-powered product is to think about the way the data is being collected and the way we're going about utilizing that data to build predictions. And one thing that's very easily forgotten and a lot of times is missed and it stops a lot of companies from being successful and a lot of startups from getting to the next level is this thinking and this realization that 80% of the work happens after the first version ships. Like any other product, whether it's AI-powered product or a regular product or an AI-centered product, at the end of the day, the name of the game is iteration. And the only way you get it to a point that people are getting value from and you get to product market fit is by iterating, iterating fast, iterating as much as you can to be able to get your model to the point that it can create value for the end users. And again, like I mentioned, the one thing that we definitely didn't do as fast as we should have is faking a lot of the ideas that you have and a lot of the different experiments you wanna run and faking that with your early users and use Wizard of Oz fake doing and existing machine learning models to validate an opportunity before expending all of your resources and time in building something no one ever wants to use. And as important it is for data science team to think of recall and precision, it's also important for product leaders and business leaders to think about weighing the cost of false positives and false negatives and the impact it's having on the end users and the overall business and how it's impacting the end-to-end product experience. And last for lessons learned, the other thing that we learned along the way is that when we are building a product, it's really important to think about how AI can augment the experience instead of thinking, you know, how can we automate this decision-making or how we can automate this entire process. The thinking should be shift to how can we augment that process? How can we enable the person who's making that decision to do it a lot better, to be a lot more effective? And how can we make them into, turn them into their superpowers? And again, it's about building better cameras, not building better photographers, not better cameras. How can we focus on building better marketers, not a better marketing tool? And again, it's a really completely different way of thinking about it because It's going down to understanding your users, what they do, how they're creating value, how does entire value creation ecosystem looks like, and how our models and how our product can enable these people that are doing these things and making these decisions to be able to do that a lot more effectively. It's really important when you're building these models, and this is again something that we learned the hard way, to have the right people labeling your model. So at CrowdRift, it was really important to identifying what a good photo is. And, you know, we had our— if you have your engineers determining what is a good photo, what is a bad photo, what is an average photo or mediocre photo for the travel industry, your models are not going to perform as well as it would if you're getting professionals who are really good at visual marketing and knowing what converts and what is a great travel photo to do that labeling for you. And again, thinking about it makes a huge difference in the end results. And machine learning, as much as it's a scientific process, it's also a very creative process and it requires a lot of creativity. And the key here that we learned is we need to include different disciplines and different teams in the process to make sure that we are building a model that is creating value for end users. And again, it's not just for engineers, it's everyone needs to get involved. And that's why it's so important for our product leaders and product designers to understand the value and the role that they play in building better AI-powered products. Last but not least, again, falling in love with the problem, not the AI algorithm. Is the easiest thing to be said, but the hardest thing to be done. It's all the time as humans, we tend to fall in love with the solutions and the models that we create and these amazing, precise model that we've built. But it really doesn't, that's not what matters. Our customers, our users don't care about that. What they care about is that how you're augmenting their experience, making them a better version of themselves. And with that, that's all I have. So I went a little bit over time, a lot of our time. But if you'd like to connect with me and have any questions, you can contact me at my email address or on LinkedIn or on Twitter.

All right, thank you so much, Amin. If you can keep the slides, I'd like to go back to those occasionally. Yeah, lots of very interesting topics for sure. And one of the things that I like about product in general is that there are a few different philosophies and schools of thoughts as it comes to product development. And it's always fun to see, you know, parts that, parts that people think about product development differently. I wrote a bunch of questions to ask you, so I'm just going to go through them one by one. Sure. In general, I feel like you are a lot more, a lot nicer than I am about your thinking about product. For example, on your lessons, on the first lessons learned slide, you said that 80% of the work starts after your first shipping. To me, it is 100% of the work, really, because when you do your first shipping, that's like— before that, nothing really exists. So like everything that you're doing before your first shipping is just getting to zero, like just getting to the start. And that's a mindset, like even the 80% is far reach for a lot of people. Like I've seen a lot of data science and machine learning teams that are just happy that they have something that is working in Jupyter Notebooks, let alone the shipping, let alone the user experience, et cetera, et cetera. And that's a mindset that is heavily involved in a lot of the things that you talked about. As you repeated a few times, falling in love with the problem rather than solution is a big part that a lot of people don't quite get. And we run product workshops, like pretty much all of the workshops that we do have machine learning as well as product development in them. And that's the part that people struggle with the most. So in your opinion, There's obviously, like, again, you know, learning from the same lesson, you know, technical people don't particularly love thinking about who's going to use this, how are they going to use it, right? So that's not where their happy medium is. So, you know, listening to the same lesson, you know, how could we set up the environments for our technical folks? So that they end up doing things in a way that is more aligned with everything that you talked about, you know, rather than what they're usually used to, which is just do the technical, don't think about the rest of it at all.

Totally, totally. No, no, that's a great point. And I think the main takeaway here and the one thing that we learned the hard way is The reason I'm saying 80% of the work happens after the first version ships is this thinking around validating our hypothesis and validating the opportunity as soon as possible. Usually machine learning roadmaps, product roadmaps do not follow the same agile feature development that you can build a feature in like a week and ship it next week. Take building a, for example, a recommender system, uh, takes a long time to get it to a point that you— it really can be utilized and it's, uh, can be in production. And, and, and turning all the models that you have into a production-ready, uh, uh, uh, code that you can ship to production. And the one thing here to thinking is how we can, as product leaders, um, and, and data science leaders, to use our product sense and use our creativity. And that's why it's a creative process to validate it faster, right? So, and this is something I've noticed a lot of Canadian companies are not good at, as good as the companies in the US. It's around this, you know, being very conservative with our models, being conservative with our product development. We don't wanna ship something unless we know that it's, you know, covering all the edge cases, it's working as expected. It's not, it's something that we are not embarrassed by. And as the CEO of LinkedIn said, famously, you know, we, the way they thought about it and the way that Silicon Valley thinks about it is that if you're not embarrassed by the first version of your product, you've shipped too late. And now I'd say this, but I'm not also quite there yet. Like I never have released a product that I've been embarrassed by, but it's something that I'm definitely trying to get better at, like better at shipping fast. Like what can we do if there is a model and there's a problem that we're solving and the solution is to build this content-based collaboration filtering model for it, how can we go about validating it? And honestly, sometimes it means that, sometimes it might mean that we don't even need to actually build a fully built functionality or a model. It can be that just we can get people to do usability testing or doing user research and discovery to see if they find any value in that. And that's part of the process, right? Always a lot of things, a lot of times people think like a minimum viable product is something that's shipped and it's in production and people are using it. A lot of times it can be literally just validating something, validating an idea or validating a solution that we wanna build without a minimum number of resources allocated to it. So that's the kind of thinking again, Silicon Valley is really good at. But in Canada I've noticed we are still not quite there when it comes to product development. A lot of times we do say that, you know, we want to ship fast and learn fast and iterate fast, but at the end of the day, things take a lot longer than they should. And by the time we bring something to the market, we just learn that it's just not the right product. And there's a really interesting book I'll share that with you after so you can share it with everyone. It's by the person who was running Google's— he was a director of Google Experimentation team, and he talks about this idea of pretotyping. So not prototyping, but pretotyping. And the thinking and the idea is that instead of prototype something and then bringing it to the market, which takes longer, we can just pretotype it. And it's even prior to prototyping it to validate the opportunity, right? The most expensive thing for any company to do and the most dangerous thing for any company or any startup to do is building something no one ever wants to use, or it's not going to get users as excited as it got us excited about it. And that's why 80% of the work really happens after the first version ships. And the faster we do that, the faster we can learn. And in the software world, the companies that learn the fastest are the ones that end up taking over the market and becoming the number one players.

Definitely. We actually had a talk a while ago from some research, from some researcher in the US who has studied a very large dataset about startup failures. And one of the very, very interesting things for me that they discovered was that there is a very direct relation, there is no direct relationship with the success rate and the amount, the amount of effort. However, there is a significant relationship with success rate. And they had this parameter representing how much have you learned from the previous failures, or like how much consistency is there between your idea and the previous ideas that you've tried. In other words, if you just fail and jump to a completely random idea, you will never— like, your chances of success will be significantly lower than if your next idea is heavily influenced by your previous failure, which is, I think, largely what you're saying here as well.

Right, exactly. Yeah, exactly.

Awesome. So, so I think what you're talking here is essentially, you know, different ways of de-risking a major undertaking like an ML project that will turn into a product. What are some of the differences? Because I think you emphasized a couple of times that there's difference between AI products and, you know, the traditional software products. So what are the differences in your mind between these two types of products that you might be building?

Yeah, for sure. So it's really interesting, and one thing that I've been fascinated by is TikTok. So TikTok, two years ago, again, when I was looking at it, and they've evolved it and they've come a long way since two years ago. But at a time when I was at CrowdRift, TikTok was, you just really getting a momentum and growing really fast. And at that time, what TikTok did really well that Instagram did not do was they built an AI-first product. And with Instagram, what they had on their Explore tab was a set of content that it would show you based on similar content that you had interacted with and had yet engaged with. And the signals were not as strong as what TikTok had. And TikTok, what they did, and they did really well, was they built an AI-centered and AI-first product where every single interaction, even at the speed that you swipe up or down, and at the speed that you like something, and the speed that you go through one and look at specific content, it teaches them something about your preferences and what you want to see more of. And it's no surprise that the average time spent on TikTok is 57 minutes a day by the users is because that they've created this continuous content cycle of showing you the right content every single time and getting you engaged with their product. And the difference there, again, what they've done really well is understanding and going back to the first principles of What gets us to spend the maximum amount of time watching TV or watching shows or interacting with something? And with building AI products, it's fascinating, but it requires even stronger and better product sense from product leaders to be able to build successful AI-powered products because then it comes to building a product that you want to utilize your machine learning models and AI models to improve the user experience and augment the experience, it always goes back to the first principles. What are the users doing today and how are they doing it today? And how can we go about making that more automated? So what are the kind of decisions that they're making day to day? What are what are the decision-making frameworks that they use or what are the decision-making mental models that they've built that can be improved or it can be done more effectively if we help, if we build a model around that. So the main difference is that. And the other thing that again, we learned definitely the hard way was that building a crowd brief, it was a combination of building our regular set of functionalities and our AI-powered set of functionalities for the travel marketers. And what we learned is that they don't follow the same roadmap. So if you, you know, you're using the same centralized project management system to keep track of, you know, the progress and the learnings and why not, they really do need to follow a different process and they do need to be treated differently. And one thing that needs to be done in the process is maximizing for fastest feedback loop. How can we get it? How can we get our data science projects to the point that we are getting feedback from end users as fast as we are shipping them, as fast as we are creating new versions of these and having a very clear transparency around it. And then the The other thing that, again, it's like a mentality that we need to change for our companies in Canada is the mentality around failure and celebrating failure. With a lot of machine learning products, we did a lot of projects that we never ended up bringing to production because they failed. They just never ended up working. And that's more of a cultural thing that product leaders need to change in the organization to tell the business stakeholders and the executives that not every single machine learning model that we're going to build is going to end up in production. So using that kind of, you know, McKinsey's 3 horizons, Horizon 1, Horizon 2, Horizon 3. Horizon 1 being your core business offerings, Horizon 2 being your, you know, more long-term offerings. Every machine learning project needs to be treated as a Horizon 2 or Horizon 3 project in a way that if this works, it's going to be a huge value add for users and it's going to create a lot of value. But we cannot expect every single machine learning model that we're building, it's going to end up in production because it's a very learning-heavy type of process and there's a lot of trial and error that requires us to really get into and learn. And we need to celebrate them. Sometimes they fail, but as long as we're learning something from that, we need to celebrate it and move on to the next thing.

Definitely. So maybe we'll take this offline, but to me, it sounds like what you described is not a fundamentally different way of developing a software product. It has a lot more nuances. There are marginally different things that we have to do. Like, we have to be a lot more careful about what data we collect. We have to be a lot more careful about what is the cycle of this data that comes to the system and feeds into the features that we have. We have to budget for a lot more uncertainty in what we are doing, et cetera, et cetera. So I feel like the— and the reason that I'm uncomfortable with the notion of AI products being so different than other software products is that people use that as an excuse to not do the product development right. And that's a fundamental problem that I have with a lot of people who are working in this space because they just want to do the cute, happy parts of it and they don't want to do the messy side of product development, which is the title of a talk that we had a few days ago. And they're using this excuse excuse to say, oh, this is so different, like, we don't understand what's happening, we have to, you know, really take our time to do this. And I strongly disagree with that point of view. And I push back on it whenever I can, because I think there are so many similarities that we can use, like a lot of what you said is de-risk, de-risk, de-risk. And if you're doing that, if you're working with that mindset, it doesn't matter if you're doing an AI product, it doesn't matter if it is a physical product.

Exactly.

Anyway.

Yeah, I definitely agree with you. I think it all comes down to exactly what you said, de-risking the opportunities you're tackling. And if that can be done, it doesn't matter if it's an AI-powered functionality or if it's a regular functionality you're building, it's just a matter of having the same mentality and applying that mentality to be able to build great products. And you're absolutely right, a lot of times, you know, we do build features that, uh, whether they're AI-powered or not, they don't end up working as expected. And some of them, uh, do end up taking as long as building a, uh, you know, a machine learning model. And sometimes they do also not work as well. Uh, but it goes down to how could we, whether it's AI-powered or not, uh, think about falling in love with the problem, validating a problem, and validating as fast as we we can so that we can maximize our chance of success.

Definitely. Uh, all right, I'm going to ask you an audience question and then my last question, which is a bit of a selfish question, but I would love to ask you that.

Sure.

So the audience question is, Moya is asking, um, in the case of the social— I believe he's referring to the parts that you talked about, social media aggregation and recommendation systems, uh, were there like, are those systems ones that can deal with short videos like TikTok, which is interesting that you also brought that up?

Yeah. So honestly, I'm not really familiar with TikTok's API or how they are aggregating your content on TikTok, but I know a lot of the destinations that we did work with, they were using TikTok as one of the channels. And the problem The problem that we learned again here was that usually building companies and building products that have any kind of dependencies can limit your success or affect your success in long term. At Cryogrief, our data, what was getting inputted into our machine learning systems and models was the photos that were being uploaded through Instagram and Google and YouTube. Attributes we were using for building our models was the location of these photos. And then guess what happened in 2019? I mean, we were lucky enough that we had raised our Series A. It was after our Series A, the investors didn't know about it after that happened. But little shortly after, like about a quarter after we raised our Series A, Facebook had the issue with Cambridge Analytica and they essentially removed the location endpoint from all of their APIs. So we couldn't get the location of these photos that were coming in, and it was a huge strong signal and attribute of the models that we were building. So that affected the entire system, right? Because then these photos were coming in and some of them that were from similar locations, we could kind of predict or predict with a 90% level of confidence that they were at that location. But for a good majority of them, you couldn't even tell where these photos were coming from. And with TikTok and any of these other platforms, if you're planning to build a product around it or if you're planning to build a system around it, is thinking about platform dependency. If that's where the data is coming from, if that's where you're building your models around, you always need to have a backup plan, a plan B for like, what if they're going to change these endpoints? What if they're going to change the way they're processing and passing me all these attributes and data about the content that I'm getting? And if they do, how that's going to affect my business and affect my product.

100%. Thank you so much, Amin. We have only 2 minutes left, so I'm going to ask you my last and sort of selfish question. But before that, I'm going to thank the audience for joining us today. Please go to ai.science to create an account and you get notified about all events like these. And you can also subscribe to our YouTube channel, ML Papers Explained, and check out all the videos that we have. Okay, so this is the last call, I guess, for the audience question as well. But my last question that maybe you can just talk about that and wrap up the conversation there is, So we talked a lot about how we can use AI in products. Well, a question that I'm very interested in, and I'm curious if you know examples, is how can AI be used for product design? So essentially, you know, one of the lessons learned that you had was ML can't figure out what problems to solve. So my question is, Can it not really? Is anyone trying to use machine learning to figure out what products to build?

Yeah, no, that's a great question and something that I've noticed with a lot of the startups that are trying to raise money and go to VCs for their fundraising. It's unfortunate, but there's a pattern happening right now that some of these VCs and investors are looking for the next stream or next round of AI-powered companies to invest in. And a lot of companies are taking advantage of that and literally adding AI power in front of any product that they have. And a lot of times it's just a decision tree. It's not even an AI model they build. It's like nothing novel and it's just like it doesn't even require machine learning to solve that problem. Anything getting from voice-based apps, like using voice for your specific type of users to detect if they have any issues like developing any problems in the future for your kids, older adults, to using AI to improve your Google Analytics performance or your Facebook advertising performance. Uh, it varies, like really goes all the way from, um, from all these different types of, uh, startups. And the, the thinking there is that if you use artificial intelligence, it's going to improve our offerings and make our offerings stronger or better. Um, but the reality is that, um, you know, 99% of these companies, they just using that and starting with ML and AI as a starting point, as a strategy, rather than really asking how big of— if this is a problem, how big of a problem is it? And why is AI the right approach to this? Why do we need to build an AI-powered product to solve this problem? Like the Gmail example that I use is a very simple example, but it's a great example of showing that, you know, yes, we can build an AI-powered product to detect if someone is meant to say attached or attachment or meant to attach a file to their email before they send it, so we notify them. So let's say this was a startup and they were like, well, ML-based, AI-based attachment detector and we charge you $4 a month and every time you send an email and you meant to send an attachment, we would tell you that, hey, you're forgetting to attach your file. Now, Google doesn't have a shortage of people in their machine learning team on Gmail to build this, but it's definitely the wrong way of going about building a product that's utilizing AI because it's an overkill. You don't need that. Just a rule-based system that looks for words like attach or attachment does just fine to solve that problem. And it's interesting, when I was talking to the director of product management at Gmail in a product management festival, he mentioned that for every 29 engineers, he has only one product manager. And the reason for that is that he's thinking product managers can sometimes accidentally come up with these set of ideas or innovative process to improve something with machine learning or with improving any process and overkill, which is total overkill and it doesn't really help Gmail to build a better product and they can be distracting to the engineering team. Now, I don't quite agree with that thinking and that entire mentality because I think it's definitely a 1 to 29 ratio is definitely not the best way to build a product-led company, but one of the reasons I'm there is that He was saying that at any time their engineering team is focusing on what's the most efficient and most effective way to solve this problem. And they never start with let's use machine learning or let's use, you know, a really fancy algorithm to do it. It's always about what is the problem and what's the most effective way of solving that problem. Definitely.

Thank you so much, Amin. Really appreciate the time you spend with us. Fortunately, we are out of time and hopefully we can bring you back in a year from now. For another talk, or hopefully soon. Thanks everyone who joined us today. Again, go to ai.science to create an account to get notified about events like this. Thank you so much.

Thanks so much, Amin. Thank you.