AI Access: Integrating Design and Technical Innovation in AI-First Products

Sam.

Hi everyone and welcome. My name is Sandhya Simhan and I'm the Director of marketing at DeepLearning AI. Welcome to our new event series, AI Access. It's a series of speaking events designed to feature directors of machine learning, product managers and other AI practitioners in the field to share their thoughts on the most critical AI trends and topics, as well as a deep dive into real world AI use cases. Earlier in January, OpenAI launched a neural network called Dall E that creates images from text captions for a wide range of concepts expressible in natural language. As impressive and useful as a tool that is, it is worth contemplating how AI will disrupt the design industry, which might be automated in the future and where designers will fit at that time Today we are excited to have Patrick Hebron, the Director of Machine Learning Intelligence Design at Adobe, join us to talk about integrating design and technical innovation in AI first products. Patrick is a designer, software developer, teacher and author. His work explores the intersection of machine learning, design tools, programming languages and operating systems and is specifically focused on the development of machine learning enhanced digital design tools. He leads the Machine Intelligence Design team within Adobe Design, focusing on designing new interfaces and interaction mechanisms for AI ML functionality in Adobe's Creative Tools. He's the author of Machine Learning for Designers, published by O'Reilly Media, as well as articles including Rethinking Design Tools in the Age of Machine Learning and the Unified Tool for the Education of Humans and Machines. In this talk, Patrick will share insights from his experience developing AI first products and offer a sneak peek into some of the ways AI will transform creative tools and workflows. A friendly reminder for our audience who have registered with the SLIDO ticket, you are listening to Patrick's While you're listening to Patrick's presentation, if you have any questions for him, login to our slido page. To submit your questions or upvote others, find the slido access link and the reminder emails we sent you. And now, without further ado, let's welcome Patrick.

Thank you Sandhya and thanks everyone for joining us for this. Very excited to be here. Let me get my screen share going Sunday. I assume everyone can see okay. Okay. Well Sandy is already taking care of some of my introduction, but this talk is called Integrating Design and Technical Innovation in AI First Products. I'm Patrick Hebron. As Sandhya said, I lead the Machine Intelligence Design Group at Adobe. We're located within Adobe Design Organization and our core charter is to try to identify opportunities for and then initiate and incubate either new features for existing products or entirely new products, all built around AI capabilities. So the team is organized as roughly half designers and half engineers in sort of the machine learning engineering side. We also work very closely with an embedded design researcher as well as several additional who work on various aspects of these projects. And the rationale for this organization of the team is, of course, there's a very large and extraordinary research organization as well as product teams and engineering teams across the company, and we collaborate very closely with them. The reason we try to compose the team in this way is because we feel like there's a real need for invention in design to happen at the interface between the UI UX and the algorithmic level. And so when it comes to getting entirely new model architectures, we of course look to the research organization, but within our team, we think it's really important to be able to stand up models ourselves, retrain on new content areas, sometimes do some architectural transformations, and be able to create a really, really close iterative loop between the visual design, the interaction mechanisms, and the true nature of the technology. So I'll talk of course a lot more about sort of how we think about this work and why we've approached it in this way. But I think sort of to get things going, a short version of it is to say that these kinds of technologies can do very, very powerful, exciting things which were not necessarily possible through existing tools, and also are very, very difficult to visualize in screen design. So we find that it's often necessary to actually engage the direct technology in the design work and also to really get sort of the right intuition for the technology and really open our minds to what is possible through it. We think that we need to play a lot. So as I go forward, I think you'll see some of how we think about this process. My apologies in advance. I would like nothing more than to be able to show you what we're actually building, but unfortunately I can't show you unreleased products. So we'll sort of talk about it in terms of process and look at some aspects of things, but at the same time we'll all have to wait in order to see some of this stuff. Okay, so I want to start with sort of a framing idea for how we approach looking at this work here, referring to Thomas Kuhn's famous the Structure of Scientific Revolutions and borrowing on this idea in a certain way. So, you know, if we think about something like gunpowder, the operating principle for it is in fact quite simple. But the implications of this technology are astounding, earth shattering, world changing in many, many Ways. And when we look at the pure operating principle, which I guess could be, you know, more or less articulated as rapidly expanding a gas in a semi enclosed space in order to launch a projectile, there's a huge divide between that and what happens with this technology. So the way I've organized this here is to say, you know, we have the operating principle, just stated it. And so, you know, this is the mechanism that performs the key task either in a new way or for the first time. In this case, that key task is the ability to launch a projectile. In many cases, this task is going to be inherited from a previous paradigm. In this case, bow and arrow being perhaps an inspiration for a gun. In other cases, I think the key task may be net new. That's not that often the case. But in the realm of AI, I think it actually has a higher tendency to be true. Going beyond this, we then of course have the direct impact. I think this tends to be foreseeable from the point of view of the operating principle and of the key task. In the case of guns and gunpowder, the direct impact is that it greatly increases the effect of an individual soldier. I think in sort of more general principle, this kind of direct impact is usually going to follow a similar shape of operating as a kind of force multiplier. What's much harder to see is the broader implications. These are in a sense, kind of beyond an event horizon. We can try to extrapolate from that direct impact, but look for the force multiplier and try to project forward. But even so, it's often very difficult to see the network effects and how this is sort of amplified. Right. So arguably, this technology, this operating principle, leads to one of the most astounding implications of all, that it enabled the consolidation of power. Things like colonialism and all of that can be very, very difficult to see from where we start. So the question then is, when we look at AI, how do we start to extrapolate that and try to understand what these direct implications are going to be as well as the broader impact beyond that? I think it's a very hard thing to harness. But if we can do that, then we can begin to think about how to shape the technology in the direction that we hope to take it within a broader purpose, within a social domain. So the operating principle for backpropagation is in a sense straightforward, at the same time fairly abstract. Now, I know that a lot of this audience is going to understand exactly what I'm talking about here. And for those who don't, I apologize for not having time to be able to sort of fully elaborate it. But, you know, the notion of back propagation is really a very singular mechanism that at the same time has really fueled everything that's been possible through deep learning. And therefore, kind of the modern phase of AI, it could be stated, as a method for automating inductive processes that utilizes the chain rule in calculus to compute the gradient of a loss function with respect to its weights. So what does this all mean in sort of a social context? Right. Well, we'll start by just looking at some of the key tasks and here limiting that to the domain of AI as it relates to creative tools. So I'm not going to try to be exhaustive here, just going to look at sort of a few key tasks. And at this phase of the talk, I'm going to actually very deliberately try to hold myself back from stemming into talking about the implications and the broader impact, because I want to sort of look at it, in a sense, unemotionally at just the level of what the actual functionality is, what the capabilities and what these key tasks are. And then, of course, we'll try to extrapolate those a bit further. Okay, so like I said, just a select smattering. So first, image segmentation. All right, obviously an extremely simplified version of the architecture here. But roughly speaking, we take input of a real image, a photograph, into often a convolutional neural network or an elaboration thereof, and the output of network via label map or another image where each pixel is an index representing what category of object is being depicted at that particular pixel in the input image. Here's one particular type of implementation of it, Deep Lab. And so on the left we have two photographs as inputs and their corresponding outputs. So we can see here that this particular pink is meant to indicate a person, and the green represents a bicycle, purple representing a dog, maroon representing cat. Okay, here we have something that is able to parse the semantics of an image, and we'll leave it at that stage for now. Going into slightly different realm, we have generative networks. There's obviously a diversity of architectures here. I'm just going to talk about sort of two basic categories. So here we have a very simplified, of course, view or, you know, basic rendition of generative adversarial network or a gan. And so there's two basic components. Here we have the generator component and the discriminator component, or the forger and the forgery detector. The generator is going to take a random vector as input and generate a fake output, a fake image from that, simultaneously we have a collection of real images, and we're going to alternately feed these into the discriminator, which is going to try to perform a binary classification of whether the particular image that it's looking at is from the real data set or produced by the generator. In doing this, the task it's performing is this binary classification, but under the hood. In a sense, what needs to happen in order for the system to be capable of performing this is that it needs to build up a kind of understanding of the likelihoods of the patterns that occur in the real data set. As it gets better at this, the generator also has to get better at being able to produce these patterns, and so the two elevate one another. Now, the random vector here is of course an interesting property because what it means is that the generator is going to be able to produce entities that fall within kind of the general possibility space as presented by the real image data set, but at the same time it's not going to really direct towards any particular output. A good example of this style of network would be Stylegan2, which produces these incredibly accurate or realistic looking faces. But of course, every time we roll the dice, we get sort of just a completely different person without a great deal of control at this phase of the output. Now, of course, I'm sure many of you are familiar with this technology and are aware of ways of introducing greater control into the equation, but at sort of the level of raw functionality, that's what stylegan and similar networks offer. Okay, now we can, we can start to introduce a bit of an architectural elaboration on top of this, which is the conditional Gamma architecture. So here we have the same basic setup, but we've introduced sort of a piece of additional information, both coupled with the real image as well as with the fake. And so that could be one of a few different things, and we'll look at a few different variations of that. But as per the conditional notion, this is going to help to condition the generative and discriminative tasks in, in terms of sort of localizing the meaning of, of the generation. So in the form of Big Gan, taking this sort of most literally, we have a label which distinguishes what category of entity the generator should produce. So Biggan is capable of 1000 categories of objects. And while any particular generation within a category is going to be sort of random in the same sense that was true of Stylegan, in this case, we can at least localize to which of those individual categories we want to generate outputs from. Next, we have a Network like Spade, where here the conditional input is a semantic map much like the one created by the segmentation model that we saw a moment ago. And here we are stipulating what category of entity is going to exist at each pixel in the output. So we don't control sort of the exact nature of the texturing of the output, but we can control at this phase the overall composition of the image and what it contains. And finally, just to look at one more, we have Cyclegan also conditional network, where here we are able to take in an image of one category and translate that into a different domain. So we have the image of a horse. We're going to output an image where that has been translated into a zebra or a scene of a summer environment translated into the domain of a winter environment. So, you know, just as we are sort of in general with Gans learning kind of the, the statistical likelihoods of the patterns that exist within a particular sort of domain of interest here we can also sort of understand what the absence of that domain looks like and elevate it, right? So we can sort of elevate the zebra ness in an image that is otherwise of a horse. Okay, so let's start to look at what the, what the direct impact of this is. So I think the first thing that comes to mind is that this begins to transform how we think about communicating ideas with machines and opens up a very new opportunity there. It changes the lingua Franco, or the language of exchange between the human and the machine to a more human centered one. So if a designer has a task like wanting to turn a chair into a bar stool on the left, we see something along the lines of the conventional approach that would happen in say a tool like Photoshop. So we have a sequence of component actions as organized by the design tool that enable us to enact this particular intent. This is of course somewhat arduous even for someone who knows how to do it. And there's also a steep learning curve to understand what this sequence is. And so, you know, even for the people who know how to do this, there's, there's a translation of media, right? You have your idea at the level of wanting to turn the chair into a bar stool, but you have to sort of drop out of your creative intent and into the language of the tool in order to enact this sequence. And so in the process you are sort of distracted from kind of the creative task at hand. And of course for other people there's no way in, in the first place if instead we can begin to utilize some of the kinds of understanding of the content and the translatability between content domains that we've seen. In looking at those basic architectures, we can begin to see a way of transforming how the user might communicate ideas to the tool. If the tool operates in the language of this content, then we can begin to offer features in which we do something like offer a slider to simply extend the legs of the chair. So this, of course, has a great impact in terms of who and what can happen within who can operate within a design tool and what they can do with it.