The Future of AI | Salesforce Research | Salesforce

This is Jane. She's a sales rep who uses AI conversationally, contextually and intuitively. Hey, clock.

It's almost seven, right?

That's right. Jane, you've got three back to back meetings scheduled today starting at eight. Would you like me to reschedule one of them?

Yes, please.

Oh, and your off site meeting with Steve Statin for later today was off, but is now back on.

You had me worried there for a second.

Don't worry, Jane. I got your back.

Yes, she's unstoppable. Like when there's an opportunity.

Tell me everything about Steve Statin.

She can quickly pull up summarization and be right on top of it.

Steve is a cio. He takes his time closing on any deal. He's very concerned about security. However, when it comes to his free time, he's a risk taker. He wants to go skydiving while playing the ukulele.

Skydiving with a what?

Ukulele.

Okay. And her AI acts as a recommendation engine, helping her get exactly what she wants across. Any type of experience?

The usual?

Yes.

Need an extra shot before your meeting with Steve?

Sure.

Okay, Jane, that's a large half whole milk split quad shot, no foam latte with whip, a touch of vanilla syrup and three short sprinkles of cinnamon. And that one's on them, courtesy of the loyalty program.

Sip up Jane, it's time to go. And the information she gets from AI.

How'd you know?

Can help Jane advance her deals on the fly. Literally. Not even language is a barrier for Jane.

So do we have a deal? Acuerdo deal?

She. Yes. Necor. Agreed.

And you, Steve, you got yourself a deal.

Great. Pull up their forecast.

Yep. The future is trending upward.

Are we good, Steve?

Boom.

Unstoppable. The future of CRM is the future of AI.

Ladies and gentlemen, please welcome chief scientist Salesforce Richard Socher.

Hello, everybody. I'm a little biased, but I think this is going to be the most exciting keynote of Dreamforce. What you just saw was a little bit of what we think the future of AI will be like. And we're actually quite comfortable making that kind of prediction because we're actively working on that.

Now.

Before we get into any of the meat, I have to do this forward looking slide. This is literally the future of AI keynote. So do not make any purchasing decisions for the next quarter. This stuff is a little further out now. You've heard a lot about the fourth industrial revolution. We've talked about different IoT devices, getting more data, having faster computers, more connectivity. And so on. But what we haven't really gotten into is, well, maybe revolutions are sometimes a little bit tricky and sometimes there's also a lot of hype. And we've seen that kind of hype when it comes to artificial intelligence a lot. In fact, we've had in the 1950s already, people predict that we'll have AI algorithms that can walk, talk, see, reproduce themselves, and be conscious of their own existence. So whenever AI makes a lot of progress, there's a little bit too much excitement too. Though what you've seen in this movie just now is actually not that unrealistic anymore. And the reason we're making predictions about three to five years out is that we're quite comfortable making those because we're actively working on all the things you saw in that movie, question, answering machine translation, smart forecasts, and so on. And so we want to educate you a little bit to take some of that hype away, but also make you feel empowered and excited about being on the Salesforce platform by knowing all the cool stuff that is going to come out. So let me tell you sort of the first three major ingredients of why we had these breakthroughs that we now see. And those are large enough data set sizes. You can now actually have most of the important patterns in your data that you need to make smart predictions. Then we have algorithms, and those are often deep learning algorithms these days. And an even more important idea is that they're end to end, trainable. We take raw data in a customer email and we predict the final outcome that we care about, maybe the answer to that email or the action that needs to be taken based on that email. And we have now fast enough computers that can actually go through all of those data sets and train these very powerful deep learning algorithms. And so what does that mean for your organization, and what does it mean for us as an organization too, of actually making this a reality? You need to think about three different groups, at the very least inside your organization to make these kinds of AI breakthroughs a reality for your customers. And those are of course, course, a research group. When it comes to really novel algorithms and really understanding the newest models that you have, then that is usually only step one. What's really important and really crucial in terms of time and effort is an engineering group that really thinks about the data sets that are getting fed in, labels the data sets, cleans them, and actually scales up the whole thing, makes it secure and trusted, things about GDPR compliance and a whole host of other issues that we won't go into too many Details here, we'll focus actually a lot more on the fund research part. And then of course, you need phenomenal product people who actually understand your customer, who understand where to infuse AI in the right place in the workflows. And in our case for cm, we're focused on sales, service and marketing. We know exactly what kinds of dashboards, service folks need to look at on any given day. And we know where to infuse AI at the right time, in the right place. Now, one thing we've also realized is that now that AI actually works, it's come out of the lab and actually into all these different production systems and it will make impact on pretty much every industry out there is we have to actively think about the ethics of artificial intelligence. So I'm very excited to later on bring on stage Kathy Baxter, our AI architect for AI or architect for AI ethics. And that is something that's really important because all the values that we have as a company, trust, equality, they will immediately become real life issues when we deploy these AI algorithms in the real world. Now, there are a bunch of different ways you can understand and slice and dice artificial intelligence. And when it comes to seeing AI as a replication of human intelligence, you can actually think about it in terms of three different application areas. That is robotics, the movement that we have in our visual cortex, then visual understanding, and the computer vision field. And what I think is the most interesting manifestation of human intelligence is actually our language capabilities. Now at Salesforce, we're a software company, so we will not talk much about robotics and we do care about vision because a lot of customer data is visual. So I'll talk a little bit about computer vision as well. But where we're really going to focus this keynote on is natural language processing. Tie it also to all the announcement we've been making at the streamforce about Einstein Voice Assistant. Now let's get started and let me show you a couple of really exciting research results from the last couple of years in computer vision. Here you see an algorithm that and its output based on an image. So in the top left you see a photo of a little girl. And the algorithm has never seen this picture before, but it's able to describe it in natural language. And as it is generating each word, we can actually analyze where it is focusing its attention on and see that it actually makes sense. So a lot of times we have issues with interpretability of algorithms. Not so much here. This algorithm actually shows you that while it is generating the word girl, it is actually looking at the girl. Sitting on a bench. While it is generating the word bench, it is indeed focusing in red on the bench, holding an umbrella in green. And it actually makes sense. Now, this is a pretty remarkable result. Even like five years ago, this would have been impossible. And maybe six, seven years ago, people would have barely accurately said that there's a person in this image. So this is really a testament to the kind of research that's been happening at Salesforce and elsewhere. We can even go as far as asking particular questions about images. So you can ask, what is the pattern on the cat's fur on its tail? And actually on the right side, you see where it is focusing its attention on and then actually focus most of its attention on indeed the right area of the cat's tail and correctly says that it's striped. Imagine when you applied this kind of technology to social media listening and somebody takes a picture of your shoe and said, ah, this piece of crap kind of fell apart pretty quickly. Now, without doing hashtag your company name now, you can actually identify those kinds of issues and trigger a service case from marketing and feedback, reach out to that customer and actually make them happy. In fact, we're seeing more and more of these visual capabilities actually making it into production. Here's an example where we work with an insurance company and car manufacturers like Toyota and others to identify damage that you see in a car. And as an insurance company, you can take with one quick photo, very quickly identify potential issues and give an estimate of how much you might reimburse somebody and so on. You can apply the same technology to count objects in supermarket shelves and help people instead of having to manually sit there and count how many products are still in stock, you can basically allow them to do this automatically. And these are just some of the visual capabilities that we've integrated in the last two years since we've announced Einstein in our platform on the CRM. Now, where I'm the most excited about is actually on the natural language processing side. I think in the next couple of years, we'll see a major transformation in how CRM is being used and how users will interact with them and how the customers of you, our customers, will actually interact with each of your companies. We'll see, I think, a future that is conversational, contextual, and intuitive. And we'll dive into each of these examples. And I'm actually very excited to, to bring on stage not just the folks who think about the marketing, but the actual researchers who are making this a reality. And so with that, I want to bring on stage Yingbo he's our leading speech recognition scientist. He's published in this field and he's actually building the systems that we're going to use to basically revolutionize both sales. Taking notes like what you see and smartnodes and Einstein Smart assistant or voice assistant, as well as a bunch of other very exciting applications in the future. Take it away, Yingbo.

All right.

Thank you very much, Richard. My name's Yingbo. I'm a research scientist here in Salesforce Research and my main research focus is in speech. So voice has already unlocking conversational experiences in our consumer lives. You could talk to Siri to get a weather update, to call an Uber or to schedule a reminder. You could also talk to smart devices like Google Home or Alexa to play your favorite music or to get the latest personal updates. So voice is amazing because it's a very natural and intuitive way for humans to interact with computers. So this opens up whole new opportunities, not only from a user experience perspective, but also from an access perspective. For example, now the elderly can speak to their device instead of typing, giving them more flexibility. This also lowers the technology learning curve. Kids can simply talk to the technology. Voice has the potential to unlock amazing enterprise experiences too. For example, if you're in sales, imagine that if you could get instant update on your pipeline or a summary on your prospect before the next meeting. Or you could update a case without even opening your laptop, all through conversational interfaces. However, the reality is adopting voice for business is different as compared in our consumer lives. And to show what I mean by that, I would welcome Tim Van Long on stage to show us. Hi Tim.

Hey, how's it going?

Yingbo?

So could you briefly remind us why exactly you're here?

You know, Yingbo, I actually ask myself that like every morning. But today I'm here to explain and demonstrate why doing and bringing voice to business is a little bit more challenging than say, asking my Alexa to play hit me baby one more time when I'm cooking dinner.

Cool.

Perfect. So let's start then. So we think there are three main problems that making adopting voice in business that is a bit challenging as compared to consumers. So first we have domain specificity where terminology varies across industries and domains.

Yeah. So Yingbo, I'm not a doctor, but I sure as heck play one here on the research keynote. And if I was a doctor, then if I was taking a voice note, I might use terminology like this. The patient's effects on the right arm are of a level where they can be released from the er. But we highly recommend they get an MRI right away. So as a doctor, I might use that language. However, if I was say, in a different industry like construction, I might use another different terminology altogether. In this situation, I might want to say something like this. The ACCMP's PSI is not at a level that can reach APHC requirements.

Wow, that's a lot of terminology. I don't know how much you guys get, but I can barely get anything. And this would give also a very hard time for a standard voice model to decipher because those are very specific terms that are particular to individual business which aren't widely used by consumers. Next, we have unpredictable audio environments too.

So Tim. Yeah.

Hey, we need a cbm. No cbm, not CRM. Oh crap.

So as you can see here, the audio environment is actually unpredictable and in a business setting, the audio environment is very dynamic. So this is a bit different as compared to speak at your smart devices, at home, in the kitchen or a living room. We can't always control where or how our customers are going to use voice technology. And in Salesforce Research, we know that quality can't be compromised when we are delivering voice service to our customers. And last but not least, we have trust and privacy that is also very, very important. Although voice is used in a business setting, it is still personal data, which means that it needs to handle very, very carefully.

Which is good news because guess what, Yingbo, I just got your lab results back from that test, but instead of telling it to everyone within voice range, I'm going to send it to you, a HIPAA compliant private messaging service.

Sure.

I really appreciate it. Thanks Tim. Not broadcasting all this in the public and these are three main challenges. Thanks, Tim. So those are three examples that really illustrate the challenge for adopting voice in business and why they are challenging. So Salesforce has a diverse user base and we would like to enable conversational interface for everyone. Which means that our models needs to be accurate across all dialects, regardless of differences in pronunciation, accents and so on. We need to make sure that we don't leave anyone behind. So Salesforce Research is bringing voice to enterprise by addressing these three challenges. First, we are building domain specific models which will then allow us better provision our model based on the user's industry and minimizing errors for words that means different things under different context. Next, we're also making sure our recognition model is accurate regardless of the conditions by normalizing the model inputs so that the noise gets minimized. By explicitly encouraging the model to learn acoustic features that are robust across a variety of type of environments. We are also augmenting our training data set by simulating voice from a variety type of situations so that the model gets to observe different types of sound scenarios. Last but not least, we understand that the importance of trust and security when it comes to our data. And this gets amplified by voice because it may contain personal or critical information. That is why we see it vital when delivering voice to our customers in a way that not only in terms of accuracy, but also making sure that security and trust are fundamental pieces in our underlying architecture. So now I would like to share with you some promising early results that we are getting with our neural network based speech recognition models. As you could see here, with the domain knowledge and expertise that we have in Salesforce with couple iterations, we are already better than a widely used sophisticated consumer oriented speech offerings in the enterprise domain. And we achieved this because we took different approaches and we believe the business setting is different as in our consumers, which also allowed us to use much less data to achieve similar or better performance as compared to others. But we are not satisfied. We are committed to make this model industry leading. So now I would like to welcome Michael on stage to show us the power of voice when it meets enterprise.

Absolutely amazing. Yingbo, it's just a pleasure to see the thoughts and the concepts that you guys are bringing through research. And it's also what makes my job so much fun. I get the pleasure of working with these brilliant minds over here on a daily basis, thinking of ways we can productize their research, their inventions to bring artificial intelligence applied to CRM across all of our customer base and allow you to extend that to your own customers. And voice as a channel is one of the most interesting ones I've come across because it's not really introducing a brand new concept, but it's the idea that we're advancing to a point that we're at an inflection to really improve the processes that you guys are going through today. And it brings up unique challenges. Right? You're not asking for the weather. As Ingbo mentioned, you guys are telling us about what's most important to your business. How you can solve cases, how you can drive new pipeline to make your business more successful. And that's a responsibility that we take incredibly seriously. And that's why Yingbo has really cemented all of his work in that foundation of trust and reliability. It's also why I'm excited to announce Einstein Assistant, which is our vision for how we can bring voice to our customers. And so, because Tim the Tool Man Taylor got to put on this cool hat. I'm gonna put on the hat, too. I'm gonna put on the role of a major manufacturing company that services their customers across the commercial real estate industry. And one of my daily jobs is to go on site visits, meet with customers, deal with property issues that might be happening. And after that, I typically would do three things. I text message my boss, I let him know the latest on what's going on, what I need to do to make sure the customer's happy. I then call some of my team members, right? I want to make sure that if someone's available, they can help solve the issue. And then finally, I call my mom, because every good boy should call their mom every day. But today I've got Einstein Voice Assistant. So let's see how that transformed my life. Switch over to the mobile, please.

Aha.

I choose a case update here. Details on my inspection for Ohana Property Group at 50 Fremont today, period. Ohana Property Group is still experiencing incremental failures on their H vac system, period. Pull the IoT logs and update the customer by Friday. Bring part number 542 for the next site visit. All right, now don't worry, before I move on with this presentation, I get to experience a very important thing right now. Transcription isn't perfect, right? We all know this. We've all dealt with our Alexa devices not playing the right song or Siri not, you know, giving us the right action item to the right task or timer to do. And the same is true with Voice today. And that's why it's so important that YINGO continues to advance and actually give us these capabilities that we can embed into the application itself. And we're not relying on the hardware device itself, and we can open up crazy new channels. So let me make a quick couple improvements here. And let's send this to Einstein. Now, the first thing you'll see, we've identified the right company that I'm actually working on right now. And we see this open case on their H Vac systems. Select that my site visit's halfway done. Now, I didn't meet with anyone here, so I don't have to choose a contact, just solo visit. And finally, I get a couple tasks that I need to perform. I get to pull the IoT logs. My whole team knows that I've got IoT logs that I need to get to the customer so they understand the latest issues. And I've got the part that I need to show up with next time I'm at my site. I click save to Salesforce, my job's done, I go onto my next site. I'm extremely productive and I'm transforming the way I do business today. But that's just the beginning, right? Because we are working on our own transcriptions engines, we can open up Assistant to any channel. So as we talk about the forward looking statements, this is what really gets me excited. We want Assistant to travel with you no matter where you're working, in the conference room, on the phone, on the go, Assistant should be there giving your sales, your service reps, your marketing agents, all the tools they need to get their job done. So my last piece of advice is listen to the rest of these amazing researchers talk about their work. Listen to Brian as he talks about deca, NLP and the Swiss army knife that he's created. How can those tools be implemented into Einstein Assistant, this new platform of engagement for your end users. And think about Veena as she talks about recommendation engines. How we can drive the next best action for your agents and your reps to complete the tasks they have at hand. So with that, I'm going to hand it off to Veena to talk about her amazing work in contextual AI. We don't get the hat.

Thanks Michael.

Everyone expects a personalized experience across different applications such as entertainment, shopping and dining. In order to provide a personalized experience you must understand your customers preferences. For example, your customers have preferences over what type of movie they would like to watch, when, how and what type of product they want to purchase, and the location and of the type of restaurant they would like to eat at all. These preferences vary and are unique to each user. Imagine if you want to watch a movie tonight, it is very important to have a movie recommendation engine which is personalized to your preference. Otherwise you will need to sift through all the possible movies to select one. But understanding customers preferences to provide this level of personalization is difficult. For one, this isn't a simple classification algorithm in a classification setting. For example, if you want to classify if an image is buttercut or not, it is very easy for you to collect some labeled training data and build a machine learning model. It is straightforward to provide such labels. Yes, this is a cat. No this is not a cat. In a recommendation setting, however, there is no ground truth label of what the best recommendation is since your customers are only shown with a small set of the relevant items. However, in general there are many more relevant items. It is impossible for you to ask your customers to label all their preferences. To make it worse, different people have different preferences. Second, there is often a lack of history. If you have an existing customer who comes to your website, then you have some historical interactions that the customer hides with the website. But imagine if you have a new customer, John, who comes to your website for the first time. You do not know what John has looked at before. You do not know what he has previously purchased before. What are you going to recommend to him in this case? Lastly, recommendation makes uses of multi channel data. In order to get a complete picture of your customers preferences, you must be able to make sense of both structured and unstructured data. Structured data can include clicks and purchases. Unstructured data may include image, text and attributes. Salesforce Research is bringing the next generation of recommendation to businesses by addressing these three challenges. To address the problem of no ground truth data, we approximate customer preferences based on their historical interactions. So if you see this or if you know I have searched for maternity clothes, then you can anticipate that I will need to buy baby clothes, diapers and formula in a couple of months. And I will be very happy with such kind of recommendation because it saves me a lot of time and I care about what is the best product for my baby. To solve the second challenge, when there is no historical interactions between users and the items, we will make recommendations based on the content. If you clicked on a new item, we will recommend other items based on the content similarity. Similarly, we can recommend items to new users based on user similarity. When we do recommendations based on content, we need to be able to make sense of the multi channel data including video, text and image. At Salesforce Research we continue to push the state of the art models on modeling different content sources. For example, in image processing we have advanced the techniques to comprehend the meaning of different texts. In image processing, we have advanced techniques to extract the most important aspect from the image. We combine these different sources in a meaningful way to provide a better recommendation engine. In a recommendation setting, we also need to keep in mind what is known as the confirmation bias problem which is only showing you things that you have seen before. For example, you might click on some pink toys before, then later all you see are pink toys. Maybe there is another building block toy that you also really like. It will not be recommended to you. It's just because it's not in pink. So when we design a new recommendation engine we need to keep this in mind so we provide a user with a more diverse result. At Salesforce Research we are passionate about bringing the next generation recommendation engine to both you and your businesses. Next, I would like to welcome Brian on stage to talk about how multitask learning makes AI more intuitive.

All right, thanks, Veena. I'm really excited to share with you some of our recent breakthrough research on something called multitask learning, which we're using to make AI, especially in the context of natural language, have a more intuitive approach to understanding meaning and the kinds of things we want it to do. What's been really fascinating about the last few years working in this field has been to see how a particular kind of AI called deep learning has started to allow us to engineer the intuitive. The things that are so easy for us to do that we actually find it very difficult to describe how we do that. And as research scientists, it's kind of our job to tease apart all the complexity that goes into an intuitive behavior. We have to dissect and analyze all of the things that humans take for granted when they do things like see or hear or communicate with language. So, for example, the way that you and I understand language gives us tremendous amounts of flexibility in how we use it. I can ask all sorts of kinds of questions, and to each one of those questions, I can give any number of responses. I can read things, summarize that information. I can translate it into other languages for other people to understand. I can even look at my phone, read a text message, and speculate about the inner thoughts and emotions of another human being. This all just goes to show how our understanding of language gives us a wide range of opportunity to do many different kinds of things. We kind of have a unified system of meaning that we use to do many different kinds of things. But this isn't really the way that AI works right now. The way that AI works right now, we typically take one task at a time, something like sentiment classification or question answering or summarization. We would take a team of scientists and engineers and have them work on designing one model specifically for that task. But during the optimization process, as we're developing that model, we typically lose the ability to do other kinds of tasks. So our sentiment classification models are so fine tuned to do that. The one thing that when we want to do, question answering, we have to maybe make a new team, design a whole new model, give it new data, and train it specifically for that task as well. And this process just continues when we want to do summarization, same thing. But with these models, once they've learned how to do a specific task, it's very difficult to get them to share knowledge between them. So a model, even if it's really good at summarization, and it's learned maybe something generally applicable, something they could use across language for many different kinds of things. It's difficult to get it to share that knowledge with a sentiment model or a new model that's learning a new task. But ideally, we would share this information again. We have this more intuitive understanding that would allow us to do all of these things. But right now, all these tasks are separate. At Salesforce, we kind of. We see the trajectory because we believe in conversational AI and we believe in systems underpinned by complex natural language processing. We see how people are asking more and more of these systems. We want to ask them to do different kinds of things. We want to interact with them and more linguistically intuitive ways. And we have over 150,000 customers, I think, each one with slightly different use cases, where if we took a research scientist and tried to design a model for each one, we would never finish. What we need to do instead is focus on building more general models, models that can do many kinds of things. But how can we do this? One of the reasons that the single task methodology works so well is because people can focus on designing models that take advantage of the particularities of a single task. Once you're trying to design a model to do, theoretically, anything with natural language, you don't get to take advantage of those things, and so you can't craft a model specifically for that. Well, the good news is, the reason that I get to be up here is we've basically done that. We've created a model, and you can take a look at the very pretty, but probably, I'm guessing, mostly incomprehensible diagram of the model we came up with. As I describe to you a bit, how we did this and how it's so significant as a methodological shift what we did. Instead of taking a single task and focusing on that, we took 10 of the hardest tasks we could find in natural language processing. Everything from machine translation to text summarization, logical inference. We even taught it how to translate human queries into SQL database queries. Because we have all these different kinds of needs in mind. We call these 10 tasks the Natural Language Decathlon or Dec NLP because. Well, partly because this evokes the challenge of mastering many different kinds of things, and partly because I just love the image of myself as a little like a coach, you know, training my algorithms and encouraging them to be the best that they can be as they face this epic challenge. And this is really what we did. We spent a ton of time trying out all sorts of different things. And the trick is, with multitask learning, if you make a change to your model that helps you on sentiment or summarization, it might actually hurt you on questions answering or translation. So what we did was we used the DECA nlp, the natural language decathlon, to guide our decisions, to help us discover which changes and design decisions would be more generally useful instead of just applying to a single task. And what you're looking at here is basically the current decathlon champion. We call it the multitask question answering network. Well, because it does multiple tasks and because it treats everything as a form of question answering. Every time you want to interface with this model, you give it some source of information and you ask a question about it. Whether it's, you know, looking for some sort of information or you want a summary, you want it to translate, Everything operates through this interface. And that's how we unified theoretically, anything in natural language processing. I know this is a little bit abstract though, but I want to highlight some of the really fascinating characteristics of this model. The first thing is not only is it able to learn all 10 tasks pretty well, if you give this model trained on the decathlon a new task, maybe translating to a new language or some funny little English grammar task, it will actually learn how to do that more efficiently and better than the models that that didn't train for the decathlon. On top of that, one thing I'm really excited about is kind of like the holy grail for AI. We're getting a little glimpse of it. The champ here can actually take a new task and we can ask it questions about things that it's never really seen before, things that it's never really had to do before. And it will give us pretty good answers a lot of time. That's because it has this more general sense of understanding, has a lot more practice with language in the general setting. But to make this more concrete, we're actually going to walk you through a live demo of the multitask question answering network. We have Michael Jones here, who is one of our directors of research, MJ Freshy Fresh to most of us. And he's going to help me walk you through a research demo. It's not a product demo. This is a research keynote. It's not even on a product roadmap or anything like that. But this is Dreamforce, you know, so dream with me for a bit and imagine that as we walk through this demo, this somehow plugs into some application downstream, something like that. We just want to show off the natural Language processing system. So before we get started, there are two inputs here. As I mentioned, this network treats everything as question answering. So the first is going to be a question and the second is the information source that we're giving the model so that it can answer our question. And we've picked out some fun use cases to show how this might transform CRM in the future. Okay, so I am no longer Brian the research scientist. Instead I am now the far more charismatic Brian the sales rep.

And I've.

Been walking around, I just closed my last deal and I see everybody wearing these weird wool sneakers. I don't really know what they are, but I hear people talking about how there's this cool company called Allbirds and I want to know why we don't have any deals with Allbirds. Why aren't they using our products? What we're gonna do is bring in the Champ here to help me do some basic discovery work. So, mj, how about we pull up the Wikipedia page for allbirds and we'll use that as an initial information source. I don't have time to read this. So instead we're gonna have the Champ do it for us and we can ask all sorts of basic research questions for this initial phase. Let's ask, for example, like where are they based? Something like that. I want to make sure they're in my territory. And here you'll see we've used the URL because this interface is just allowing us to pull out that text which we have listed down here at the bottom. This is the context that is pulled from the Wikipedia page. We see the question that we asked and we see an answer. San Francisco. Perfect. That's where I work. These guys are in my territory. Now we can ask more abstract things as well. I'm a sales rep, but I'm also a decent guy. So I kind of want to know, you know, what's their mission or what's their goal? You know, is this a company I can get behind or, you know, am I just going to be going for the money? Let's see. So the Champ here says that Allbirds likes to design environmentally friendly footwear. Alright, I can get behind that. Seems reasonable to me. But if we fast forward in time. Now let's imagine I did take on this case and been working on it for a while, but I'm busy as always. I have lots of different accounts, lots of different clients. I've kind of gotten a little behind, you know, with Allbirds especially I. It's been like a month since the last time I checked on things. So how about we pull up a press release from Allbirds, something in the last month, something recent, Anything's good. So let's see. Yeah, Passmont's good. Something about their stores. Again, we can do the same thing. We'll pull out the URL, paste that in as an information source for the decathlon trained model, multitask model, and let's just ask what the summary is. So I'm busy. I don't want to read. I don't like reading. Again, here's the context. And it looks like they have stores in San Francisco and Washington, D.C. in the works and they're going to open eight more stores in the U.S. okay, so I've been slacking off, but now I know I really need to get back on top of this because these guys are really expanding quite quickly. This is an example of the summarization and I could use this to filter all sorts of kinds of information out. If you fast forward one more time, let's imagine now that I've been working pretty hard and I think we're in the end game. I've got this objective that everyone should be in agreement and by the end of the week we want to close the deal. So what we can do is we can actually give that to the model as an information source and then we could give it other kinds of information like emails. And we can ask the decathlon model to read those emails one by one and see whether they're in contradiction with my objective or maybe if they're in line with it, or maybe if they're just totally irrelevant. So if we paste in an example email something like, okay, like the CIO has additional questions and they require customer preferences. They think there's still a lot of work to do. And we're going to ask the model, okay, is this a contradiction or not? As you'll see, the model actually gets this correct. It's pretty good at these kind of logical inference tasks, even at the same time as it's good at doing really flexible translation tasks and question answering tasks. This is great for me because this, you know, this can raise a red flag. I can know, oh, these things aren't in line with what I'm thinking now I can spend more time on that. These are some of the examples of CRM specific use cases that we thought about. But I kind of want to step back and go back to the Holy Grail situation I was talking about before. What we're really looking for is zero shot generalization to some degree, applying things that the model has learned to new, novel situations. So for this last example, going to have MJ here come up with some story. He likes to surprise me. And what we'll do is we'll just ask it some questions, some things that aren't in line with anything we really trained it to do. And MJ's always the comedian. And then. Yeah, so here, this is a good example, because this model doesn't have anything. It doesn't know anything about keynote presentations. I don't. We never trained it to know how. Like clapping. I don't know. So let's ask it. Like, something like, okay, am I sad or am I happy? It's reasonable. Let's see. Okay. Yes, I can actually verify that. I will be very sad.

Yes, thank you.

I'd be very sad. And I'm not sad. I'm quite happy. Thank you, guys. I have one more slide, so keep that in mind at the end. I want some applause, too. Let's go back now to the slides. Demo's over. What you just saw was an example of how this model was able to do a lot of different kinds of things, but that was actually still just a subset of the things we thought about and the things that this model was trained for. Like I said, it could maybe translate things into SQL for you. It could translate into Germany, maybe other languages in the future. What I'm really excited about is to see how this changes the actual development cycle of AI. Remember how I was talking about how all the models were separate and it was hard to get them to share knowledge? Well, now, because we have one model, hopefully we can have all those teams combine their efforts and share their knowledge and share their experience. Not have them restart every single time, but feed into some base system that we can later adjust for very specific use cases. But, okay, so one thing to keep in mind with 10 times the data and 10 times the fancy abilities and 10 times the tasks is that we really have 10 times the responsibility when we're deploying this kind of model. Make sure it's free of bias, consider the ethical implications. So what I want to do is bring back on stage our chief scientist, Richard, along with the architect of ethics and AI at Salesforce, Kathy Baxter. And they're going to walk you through some of the implications this kind of technology will have for the future.

All right. You could see the amazing breakthroughs that have been happening here at Salesforce Research. And as the gap between the fundamental basic research and real applications for our customers is getting smaller and smaller, and the speed of These improvements gets faster and faster. It is very important for us to bring in an ethical mindset right away as we're doing the research and thinking about how to get it into our product. So I'm very excited to have Kathy Baxter with me today, our architect for ethical AI practice. And I want to ask her, kathy, what is your role? Why are you here with us?

I absolutely love my job. I have the best job. I know you probably think you've got the best job. I've got the best job. I get to really focus on three key areas. The first is our employees being able to work with and collaborate with the most amazing research scientists possible, thinking through how our models are impacting society and how do we build ethics into it. And then of course, working with our product engineering and user experience teams to think about, then when it becomes productized again, how do we think about the ethics of it? And the second would then be our customers, all of you. So being able to speak with all of you, understand what your questions are and think about how do we empower you to make sure that you are implementing our products in the most ethical way possible, providing training, doing customer advisory boards and user research with you. And then the third is what I like to call the world, getting to work with amazing experts and outside groups like Principia, Berkman, Klein center for Internet and Society to really continue bringing in outside perspectives. We don't want to create an echo chamber. It's really important that we continue getting additional perspectives, learning, developing our own point of view, and then of course, sharing that background with the world and of course, publishing everything that we're doing.

So help here, our audience. If you're thinking about a research group, how could somebody working in the AI practice actually influence their work? And how do you help our researchers go about doing their work?

Yes. So the, you know, one of the things that's really interesting about what we're doing is thinking through the different pieces of AI. So there are the models, like what we've been hearing about today. There's the training data, and then of course there's how our customers implement it. And so thinking through what are the societal and ethical implications, for example, voice AI that we've been thinking about or hearing about today. We know that voice recognition simply doesn't work well enough for women and people with non Midwestern US Accents. You might have been able to tell. I have a little bit of a Southern twang. I'm from Georgia. I tend to have a great deal of difficulty with my smart devices, understanding me and my twang. And so we.