World's First Artificial Intelligence CRM: Einstein Keynote | Salesforce

Hello, ladies and gentlemen, how you doing? My name is Marco Castellana and I have a cool job. I'm VP product of this awesome team, the Einstein team, that is building cutting edge technology every day. Every day we're inventing something new. And so I'm really excited to show you guys today some of the stuff that we've been working on that you can use to add intelligence to your own applications. But before we begin, the purpose of artificial intelligence is to make forward looking statements. But if we make any forward looking statements today, please make your purchasing decisions based solely on on present functionality. Now here we find ourselves in the midst of the fourth Industrial revolution. As Keith Block mentioned yesterday in the keynote, intelligence is changing everything. It's changing how we interact with everything and with each other. And it's changing what your customers expect of you. And whose job is it to bring the businesses, the governments, the NGOs that you represent into the fourth industrial revolution? It's all of you. It's your job. That's right. And we would like to thank you for allowing us to join you on this journey. Now, the best kind of of artificial intelligence is the kind that feels natural. I was playing with Spotify the other day and I came upon this new intelligent feature that they have called the time capsule. And it picked on the top there a song called Even Flow by Pearl Jam. I used to play that song over and over again. I just repeated it over and over again on a cassette tape in 1993 as I ran my fingers through my thick, lustrous hair. And Spotify was intelligent enough to know this. I don't think I ever played that song on Spotify, but somehow it figured that out. That and a whole bunch of other songs that came from my past. So you're finding intelligence in music applications. You're finding it of course on map applications and even in fantasy football. It's starting to recommend intelligent picks to you to trade for. So it's all over these consumer applications, but businesses have been a little bit slower to adopt it. Why is that? Because it's hard. Traditionally, it's not been that easy to add intelligence to your business applications because first of all, you had to start by gathering all your data from all these disparate places, data warehouses and stuff, and put it in the same place. And then you had to use some artificial intelligence toolkit, like for example, the open source system R, and you had to know how to use that. You had to know how to use the algorithms, when to use linear regression, when to use a neural Net when to use a random forest, which is actually a real thing, a random forest. And then you had to take whatever predictions and stuff you were making and put that back into the applications that your users are using. And that was often the hardest part. But Einstein is not a general purpose artificial intelligence toolkit. Einstein is AI for CRM or more broadly, AI for the Salesforce platform. Well, what that means is that first of all, when it comes to the data, the data is already here. All of you have already been entrusting all of this data to Salesforce for many years. The metadata is here. Just by virtue of setting up Salesforce and customizing it, you, you are specifying metadata. This field here is not just a bunch of numbers, it's a postal code. This field here is not just a bunch of strings, it's an email address. And Einstein uses that to learn from your data. That metadata is very important to make this whole system work. And the app is here. Salesforce is the application that your users are using. And so it makes it much easier to expose the output of your artificial intelligence to your users. Now, there are three types of questions, broadly speaking, that you can answer with Einstein. The first type is descriptive. Why were my sales in California so good last quarter? It can tell you answers and give you insights about that. Then you can make predictions with it. Who is going to attrit? Which of my customers are going to attrit in the coming month? And finally you can do prescriptive what if scenarios. What if I added more contact center agents to my tier one support? How would that affect my CSAT number? For example? These are the kinds of questions that you can answer with Einstein, which is a first class member of the Salesforce platform. Just like lightning, just like Process builder. And now with my Einstein, which we're going to show you today, you can build your own custom intelligent applications. You can add intelligence to your own applications. So broadly speaking, Einstein applications fall into three buckets. There are a set of out of the box AI applications. And every Salesforce cloud now has an Einstein associated with it. So every Salesforce cloud now has intelligence. In sales cloud Einstein, for example, that means prioritizing leads and forecasting where you're going to end up in the quarter. In service cloud, it's about tier one triage. In commerce cloud, it's about recommendation. But every cloud has pre built Einstein applications and intelligence baked into it. Then we have declarative, declarative applications. And this is where you can build your own custom intelligent apps without writing any code. And that Includes Einstein Discovery and what is now in pilot, Einstein Prediction Builder. So that's going to be a fascinating thing we're going to have a look at today. And finally, for those of you developers here, we have programmatic APIs by which you can add some deep learning goodness like vision and extract useful information from unstructured text by writing just a little bit of code. So I could stand up here and talk about this stuff all day long, or we could show it to you. Which will it be? You want me to keep talking? No, I'm kidding. All right, let's do this. So to give you a look at the out of the box applications I'd like to bring to the stage from Einstein Product Marketing, the fearless Frenchman Julian so much.

Thank you, sir.

Bam.

And thank you, Marco. Welcome, everybody. Good afternoon. Very happy to be here. We're gonna do a quick show of hands. I want to know who believes that ultimately AI has the potential to give you a competitive advantage? Wow, that's pretty massive, right? A lot of people. Cool. Who has a firm AI strategy in place? Not one. Okay, one hand. This is Marco only Marco has a firm AI strategy in place. Listen, this is exactly what we see, right? So customers know some things out there. There's great potential for AI, but it is just hard to have a strategy in place. Why? Why is that? It's because you all, including us, are like these little characters in this slide. You're still searching, you're still looking for use cases, KPIs. Do I have the right data? I don't know where to start. It's just too complex. And we at Salesforce believe that the right way to help you get started is with out of the box apps, what we call the flip of the switch AI. And this is what we really started shipping when we launched Einstein about two years ago. Now with apps that really fuel the success platform, the customer success platform of Salesforce, starting with marketing cloud Einstein, all the way to sales service, commerce and community. So if you take marketing cloud Einstein, for example, we have awesome features like Einstein engagement scoring that would predict with machine learning models, the likelihood of an individual to click, buy, subscribe, unsubscribe. Pretty cool, right? And that goes all the way to commerce cloud and others. If we talk about Saleslod Einstein, I have one slide on that. Saleslod Einstein. You've seen that product a few times. What is new is that we're now, to Marco's point, bringing the power of analytics and knowing what happened in the past to the future looking stuff of AI. So not only can we boost deal win rates and lead conversion rates with Einstein machine learning and AI, but also can we discover the trends and take actions of that with our sales analytic app today, embedded with the same product, Salvador Einstein. Salvador, Einstein has. Now. I don't have enough fingers. Yes, I do. Nine features. Nine features of Einstein embedded in one single product. The last piece of it of which being all the productivity tools, things like Salesforce Inbox or things like Einstein Activity Capture. Right. And forecasting is coming soon. So this is a very exciting product for us and for you. But again, as Marco said, enough slides. Why don't we now go ahead and see the product in action? And to do that, I'm going to start with the point of view of the admin. How many people are admins or used to be admins in the audience? Okay, that's a pretty good stat. I hope you're going to appreciate that we made that super easy for you. Three simple steps to get started with Einstein today. The first one, get educated, take the trails. So we made that, again, easy and we just give you the right amount of knowledge and information that you need. We don't want to, like, overload you with AI stuff because it's not the point. Back to Marco's idea. We don't want to talk about algorithms. We want to make that super simple. And we're being pragmatic here, so. Okay, cool. Now you took some trails. You're ready for step two. Step two is really something that I love because I believe that sometimes as admins, we set up stuff that ultimately does not fit well with our instance of Salesforce. Right. Let's be frank. That happens sometimes. Well, now with the readiness assessor, we're able to assess whether we have the right data and the right processes in our orgs before we even set up Einstein. And we do that with one flip of a switch. We start the readiness assessor here, and what that gives you is a rich PDF report where you have the breakdown for the nine. Yes, nine features of Cephalot Einstein. It gives you a green, yellow, red color for each and every one of them. And now you know, what are the good ones for you and what are the actions for. For the features that are not a good fit for you to have them become a good fit ultimately. And by the way, if you want to run that report yourself, you just go one floor down at the Einstein outpost. How many people have seen the Einstein outpost? Yeah, a bunch of people. Cool. Well, if you haven't seen that or if you want to come back? Then please come back and we'll run the readiness assessor in your org.

Cool.

That was step two. Step three, it's time to set up. Right. So you're going to love this because the setup is brain dead easy. It's just the flip of a switch that we mentioned. So we turn on Einstein with that setup hint link here and that automatically turns each and every of the nine Einstein features in your org. This is where the magic happens. But you don't see that it's not in the ui. What Einstein does is it's going to take pull some data, build some models, do all the AI magic and the machine learning and then writes the scores back to, in this case, sales cloud so that then the next day if as a sales rep. And now I'm going to switch gears a bit and I'll be in the sales rep or the VP of sales character. I see a bunch of Einstein insights and a bunch of Einstein scores right here. Magically in my ui. I didn't do anything. All my admin did was these three steps the day before. So I see the Einstein insights, which are really personalized to me as a sales rep. If Marco was to log in, he would see different set of insights. That's cool. And if we go to the opportunities tab, let's face it, we sales reps typically work. The majority of our time is spent on the opportunity object. Right? This is really the red and butter of sales reps. And what we see here appearing is a mini analytic dashboard. Doesn't it look familiar? Feels like you've seen that kind of dash already and wait, it's actually dynamic. So if I go ahead and click on one of the links of one of the mini widgets, that looks even more familiar because I can slice and dice and do some drill down exploration and change the chart type and customize that. That's right. This is Sales Analytic app. As I said, it's now one of the nine features of our sales Einstein offering. Now if we scroll down a bit, we see something that some of you might have seen already, the Einstein insights for the opportunity. So I'm not going to spend a lot of time here because I want to finish my demo with something new that I don't think people have seen yet. To do that, I'm going to go ahead and click on the forecasts tab. Let's take a look at that messy data table at the bottom of the page. This is a bunch of opportunities, right? So as a sales rep, I have no idea where to start. But we now have the new Einstein opportunity score right here. I can do that to sort my table by decreasing scores. And if I hover over the top opportunity, I now know that I need to focus on it. And I know why Einstein is never a black box. Einstein always tells you why a score or why a given insight is what it is. That's pretty cool. Okay, let's now do some simple math together. It'll be simple. So if we take that score for one given opportunity, multiply that by the amount. I'm not asking you to actually do 98 times 100 something. I'm just trying to conceptualize that. Right, so score times amount, that gives you the projected value of a given opportunity. Okay, then if you do that for all the opportunities and you sum that up, what does that give you? Forecasts. That's right, it gives you a forecast. And this is what you see at the top of the page. Einstein now has a pretty forecasting engine right here where I can see the breakdown by sales rep. I can see the breakdown by month. So I'll just go ahead and click on the month of April. It's coming soon. Right? So we'll see the forecast is at 868k. I can see the confidence interval that Einstein gives me. If I scroll down again, I see the. Yes, as we said, Einstein is not a black box. So we see the deals that moved in and out and we also see again the top predictive factors that are driving that forecast for the month of April. So if we summarize, we did the setup with three easy steps and then the experience for the line of business user like a sales rep is extremely compelling because it's all integrated within the context of your salesforce Einstein instance. But we need to move on to the next topic and the next section, the declarative AI apps. And the reason is that we listened to you and you guys have way more use cases in mind than we will ever be able to deliver with out of the box apps. And that is why we know and recognize that you need custom AI. You need to build your apps yourselves. And there's two ways to do that. And with this one, we'll use the declarative way to bring AI to your apps. And the person who will walk you through this section is my friend and colleague, Ayari Selassie, who will come up on stage. Good job, Ayari. Hi. Thank you.

Thank you, Julian. Sales cloud. Einstein is amazing. And, and I'm sure you are all very amazed and fascinated by those new capabilities. But I'm here to talk to a special group in this room. I'm here to talk to the admins, the people who want to take artificial intelligence and transform it into insights for their business using any custom object, any standard object, and customize it for your business process. Now, I've been at Salesforce for 10 years. This is my 10th year at Salesforce, and it's been an incredible journey. I started as an admin, and that journey has been so incredible because of you all, because of this incredible community that I've been a part of. So I want to thank you, thank you all for allowing us to be a part of your journey to success and creating more innovation. That being said, let's pop into this. So you all have told us and given us lots of feedback of what's important for your business. And we've heard one key theme across all of you, and that's that you want to be more predictive about your business. Whether you're in sales or service or marketing or it, you want to be able to predict what's next. Now, if you're a salesperson, you want to be able to predict which opportunities are going to close so that you can see how it's going to impact your forecast. If you're in service, you want to be able to predict, hey, what's the right agent to assign this case to so that you can route your cases more effectively. But what if you're in marketing, you want to be able to start anticipating the ROI on those email campaigns before you ever click send. And for the retailers out there, you want to be able to recommend that perfect product for every customer every time. But what about for it? Any it in the room, it in the house?

Yes.

Yes. You all have a very important job. You need to be able to identify systems end to end and how to make those business processes and applications more intelligent. It's a very, very difficult task and we want to make that easier. And that's why we built Einstein Prediction Builder. Einstein Prediction Builder is as simple as point, click, predict. And any admin can build an intelligent application, can build an AI model fast with the power of machine learning, without a PhD without a data scientist, you can build artificial intelligence for any custom object, any standard object, and any business process. And you can take those predictions because it's just a field inside of Salesforce. And you can embed that prediction in any application, in your lightning components, in your workflow, in your reports and dashboards. Amazing. And so how does it work? Well, let's talk about it. It's as simple as selecting the object that you want to build your prediction on. And Cody has a little pro tip up here. Find a repeatable business process, something that you can report on and that's where you're going to find what you want to predict. Think about your KPIs, think about what are you reporting on on a regular basis. And then you just select the fields that you want to include in your prediction. You can deselect fields just with as simple as checking and unchecking checkboxes. And you can customize that model for yourself and then you save that prediction to a new field. Everyone's built a custom field before, right? These predictive fields are just as simple as creating a regular custom field. And then Einstein does all the heavy lifting for you. Einstein applies machine learning algorithms using auto machine learning and creates a perfect model just for your business. And then you can take that prediction and you can embed it across the Salesforce application in your dashboards and your reports and your analytics, your applications, and create intelligent applications. But seeing is believing. So let's pop into a demo. This is my favorite time. Demo time. All right, I've got Katie back there on the ones and twos and we're going to talk about this incredible organization, College Forward, and how they've leveraged the Einstein prediction builder to build some very special application for you. So College Forward has an incredible mission and that mission is to help high school students get to and through college. And they have an incredible success rate, 90% success rate, with achieving this mission with their students. And they're able to do that in part by the power of the Salesforce platform. They use Salesforce to track every interaction with their students and they've customized Salesforce to their business process. As you can see, custom objects across the top students, schools, semesters, Salesforce platform. And they're using the power of Einstein analytics to track all that information. Which students are still enrolled and which ones have dropped out. It's like looking in a rear view mirror to see current and historical information about student performance. But what if College Forward had the power of a crystal ball and could look into the future? What would they predict? Well, the first thing that they'd want to predict is the likelihood of a student to drop out. And so we're going to jump in and build these predictions. We're just going to call it dropout risk. I'm going to go ahead and click next. And now Einstein is guiding us through this process. We're going to select the right custom object. In this case, we've got the student object already created here. And click next. And then we're going to select what we want to predict. Dropout. We're already tracking that in the system, and that's exactly what we want to predict. Now, Einstein is your guide, so it's going to guide you through this process and show you what these predictions might look like. So, you know, if you're thinking about it in the right way, this is perfect. So let's keep going. And now no one knows your business better than you. Oh, actually, now you're going to save this to a field. This is where you're going to have your predictive field, but no one knows your business better than you. And so this is where you have full control to customize these models on your business process. You can unselect fields that you don't want to include in your model, such as personally identifiable information or anything that may be of a compliance concern. You have full control over what's in and what stays out of these models. So this looks good. We want to include all of this information for this example. And now you gave us lots of feedback throughout the pilot and with lead scoring. And you told us that you want more control of these models. You want to be able to see segment these models using things like your record types or customer segment or your region. And so we gave you that capability by just selecting the field that you want to segment based on. We're not going to use that for this example, but the great thing is that it's there. And again, additional customization. But we're not going to do this. We're just going to jump ahead and build this prediction. All right, Einstein is going to take all this configuration, take it back into the laboratory, and it's going to build a prediction in 24 hours and it'll be made available to us to deploy. But how do we service these predictions to our end users? It's as simple as jumping in to the Lightning App Builder. We're going to hop into the Lightning App Builder, we're going to identify our component Einstein prediction builder, and we're just going to drag it onto the page. And just like that, we've taken an application and we've made it more intelligent. We've created a smart application. And now we see our scores presented on the screen. Incredible. This student is at a low risk of dropping out. And Einstein is telling us why. Factors like the gpa, the living conditions, the attendance of the student. Einstein figured these Patterns out by looking at the data, you didn't have to lift a finger automatically. Amazing. But we don't stop there. Counselors at College Forward have to manage lots of unstructured data as well. Things like notes.

Right.

And so these notes have important information as well. And so Einstein can read that as well and identify signals in that information. And here we're going to take this note, put it into Salesforce, and in just a moment, we're going to submit that information and see these things change. Now we see that the troubled living condition flag has been set and our score has been recalculated to take in consideration this unstructured and structured information. And not only that, it's not enough to just see the prediction. You have to be able to act on the prediction as well. And now we see an intelligent workflow is fired and created a case automatically so that the right counselor can follow up with that student and family. Now that is incredible. That's the power of intelligent applications. But it wasn't just Einstein behind this. It was trailblazers behind creating this application. And so with that, I'd like to invite Austin and Ashley from College Forward to come up and tell us how they built this amazing app. Coming up, guys. Awesome. Very impressive. So, Austin, I didn't have a chance to talk too much about your mission, but please tell us a little bit about College Forward and what you do.

Sure thing. So College Forward's mission is on one level, really simple but infinitely complicated and crucial to our society. So our mission is really to coach motivated, underserved students to achieve the benefits of higher education and and a college degree. The way that we go about doing that is we pair what we call near peer coaches. We have a staff of about 130AmeriCorps members who are recent college grads themselves. So they've just finished this process. We paired them up with the student, and we really make sure that everything that student needs is knocked down. Every challenge as a first generation or low income student that they might run into is addressed from the start of the finish line all the way through completion and into the workforce.

That's amazing. Incredible mission. And so what were the predictions you wanted to build and tell us how you thought about this process?

Well, we started simple and that was the really, I think, critical thing in terms of what we've learned from this process. The fact that as a small nonprofit in Central Texas, that we're up here on this stage now, I think speaks volumes to the logic that Einstein has.

Been able to do.

In so many ways this organization, Salesforce, has democratized access to really cutting edge technology for organizations like us with a social mission. So for us, it's really about how do we become more proactive as opposed to reactive in providing services to our students. Typically, when we're trying to go through this and we see a student is at risk of dropping out, it's often too late for us to do anything about it. And so the demo that you just displayed there, with the auto creation of cases and even potentially solutions that have been proven in years past, is a way for us to start really flipping the script, being proactive, meeting our students where they are and putting out these fires before they turn into real crises.

You mentioned a key word there. You said proactive. And now organizations, I'm sure lots of folks in the room are wondering, how can they be proactive about their system so that they can be ready to adopt technology like this? Can you tell us about your process and what you did to be ready for this?

Like most organizations, the way we track and manage our data hasn't been exactly, exactly the same over time. Your business model changes, things change. So to prepare to launch this amazing technology, we really had to do a deep dive of our data. So looking at our data validation rules, looking to see what are the completeness of all of these incredible fields that we want to include into this prediction. So by doing that analysis up front, setting up your systems for success, putting in data validation rules in place to protect the integrity of your data is a great way to get started so that when Einstein hits your salesforce. Org, you're able to implement it and you've got your data ready.

Thank you for that information. Now all of our trailblazers, if you're not ready yet, you know exactly what to do to get ready. And so what are you going to build next? What's coming next?

Well, I think the real future of this is just it's pretty hard to wrap our minds around. As Parker said earlier in the keynote today, it's really just limited by our creativity. The number of predictions and risk factors that we hope to build out with the Einstein team are numerous. Too many to count here. But really where this is going, my dream and vision for this we saw, if we can create a universe where Spotify can predict Marco's obsession with Eddie Vedder, why can't we enable and put the power of technology and data straight back into the hands of our students? So College Ford is an in between, between our mission and the path of success of our students. But how can we figure out ways to empower them? One of my dear colleagues for the past 10 years, Scott DelRossi, once said, the only population scaling quickly enough to solve this problem is our students themselves. So how can we start pushing these analytics and these solutions straight back into the pockets of our students?

Awesome. And what about the rest of your mission? I know that you all have an incredible plan to scale this.

So College Ford is not only a customer of Salesforce, we are also an ISV partner. Assume everybody knows what an ISV is. So we have an app on the App Exchange, and we have since 2013. So through this technology, through what we've built for our own organization, we've actually scaled that and are now serving 35 organizations across the country that are collectively serving almost a quarter of a million students. So as we're building out this amazing technology at College Forward, as we're thinking about predictions, if we can identify what is core, what we can include in our managed package, we can enable these organizations to take this technology and really run with it. So when I'm thinking about Copilot and the future of this, that's where I see it going. And I think that is just going to make it even more exciting.

So impressive. Thank you so much. Thank you, College Forward. This is amazing. You've inspired us all.

Thank you.

You saw it, you heard it to and cross incredible trailblazers in AI. And now you know everything you need to know to get started to become a trailblazer in AI as an admin with declarative applications. Now, for all you developers out there, we're ready to move on to the next chapter. And to do that, we've got the incredible Z who's going to talk about programmatic applications.

Thanks, Ayuri. Hi, everyone, My name is Zinev Laraki. I'm a product manager on the Einstein team. And I'm so excited to be here with you guys and show you how Salesforce is making it very easy to be able to build smart applications like we just saw, to be able to automate mundane tasks and build smarter applications with Einstein platform services. Sorry about that. But first, let me tell you, give you some context. So, as you might be aware, data is key to the success of AI. And the reality is we're generating so much data. Whether it's through our applications, our mobile phones, our connected devices, we're generating so much data that we can use. Now, right now, most of the data that you have in Salesforce is what we call structured data. And that's because it works beautifully with with workflow triggered applications. And a lot of the products that we just saw also deal with structured data. But the reality is, most of this new type of data that we're generating is what we call unstructured data. And by that we mean image, text and voice. And this unstructured Data is growing 40 to 60% per year. Now, unstructured data such as text, you can think about emails, chat logs, service requests, historically we haven't even been analyzing them. And image and voice, which has incredible insights, right? We haven't even been storing them because we couldn't derive value from this type of data. But now we're entering a new era where this type of data is more important than ever before because we're able to derive value from this type of data. And thanks to deep learning, we're able to understand the more complex patterns and really automate mundane tasks and gain business insights where we just couldn't before. Now this is great, but as Marco was mentioning earlier, it's hard for enterprises to implement deep learning. And the great thing is that Salesforce is making it very easy. We're basically making it easy for you to turn this unstructured data into something that you can gain value out of. So how do we do that? Well, we make it easy for you to build smart applications using Einstein platform services. And these services allow you to derive value from text and images. So, so there's two categories of services. There's Einstein Vision and Einstein Language. Under vision, we have Einstein image classification. And what this allows you to do is categorize images based off of categories that you decide. So you can think about how a field rep in the field is able to take a picture of a part and have it automatically identify what that part is and use that to automate reordering, for example. Then we have Einstein object detection. And this allows you to identify the quantity, size and location of objects within an image. And consumer good companies are loving this because they can now take a picture of an aisle and have it automatically analyze what's in the picture, identify the products and count them. And they're using that to streamline retail execution. On the language side, we have Einstein's Sentiments, which allows you to understand is a text positive, negative or neutral. And you can think about how you can use this to prioritize cases. For example, and finally, we have Einstein Intents, which is what we saw being used earlier in the case in the note that was entered. And it allows you to understand what category a text belongs to. Based off of the categories that you define. And we're actually going to go ahead and build that model together today that was used earlier in the demo. So I've been mentioning that this is, you know, we make it very easy. Salesforce has democratized deep learning. How have we done that? Well, really what we've done is we provide these services as a managed service, right? So it's a hosted infrastructure. You don't have to deal with any of your scalability needs. We take care of all of that. We make it very easy for you to use. And then also it's built on the Salesforce platform for the CRM use cases. And finally, we've been able to really close the gap between the latest advances in AI and taking products to market by really working with the Salesforce research team, which I like to say is basically half of my team. So with that, now that you know what Einstein Platform Services is, let's go ahead and talk about how do they actually work? I'm first going to tell you, and then I'm going to show you how they work. Okay, so the first step is to define what you want your model to do, right? Then you say, okay, I have to collect my data. And if you're one of those lucky people out there, you might have your data in Salesforce that's correctly labeled and you can use it right away. Now, the reality is very likely that's not going to be the case. Right? But don't worry, it's really not as big of a hurdle as you think. You often need much less data than you think to just get started. So just go for it, start collecting data, and I'll show you how this works. Then the third step is to upload and train your model. This is actually really straightforward, and I'll walk you through these steps. The fourth step is to test how your model is doing. So we provide you with model metrics so that you can understand how your model is performing and improve on it. And then once you're happy with your model, you go ahead and integrate it into your application. And this can be a Salesforce application, as we just saw, or your external application can be a standalone iOS app or Android. So now that we've covered the steps, let's go ahead and jump into the demo. Let me show you how we actually built this stuff. So what you saw earlier, right, what Yuri showed us, is text being entered in a text box and have it automatically identify what category this text belongs to, and then a case created with that type. So here what you can see is the first step that I was talking about. We identified what are the case types that we wanted for our model. In the case of College Forward, we had troubled living condition, service referral, tutoring referral, and medical referral. So once we've identified this, our next step, our second step is to collect the data. So what we did is simple CSV that you can see here. And we wrote down a bunch of examples for each of these categories. So you see the examples, or what we call in AI speak, utterances on the left and the categories or what we call labels on the right. And you'll notice we have only about 20 examples for each of these categories. So with that, the next step is to upload and train your model. Now, I'm going to show you the Einstein intent APIs, and to do that, we're going to use Postman just as the UI to help us with that. So the first step is I'm going to upload. So you see here, I'm making a post call. To upload, I'm going to select the CSV that we just saw as one of the parameters, click Send here. And as soon as I do that, I get a data set ID and I'm going to go ahead and copy this data set ID and use that to train my model. So I'm going to go here, make a post call to train my model. I specify the name of the model, so collegeforge flags. I'm gonna specify the data set that we just uploaded and I'm gonna click send. So as soon as I do that, I get a model id and this is what you're gonna be using to understand how your model works, as well as get predictions on new text that the model has never seen before. So training a model can take a few minutes to a few hours, depending on how much data you have. For the sake of time, we went ahead and trained a model just before this using the same data set. So you're going to see I'm using a little bit of a different data set here, a different model ID that you can see here. But basically it uses the exact same CSV that we just saw. So I'm going to go ahead and click Send to see what are my model metrics, how is my model actually performing? So there's a few interesting things that you'll see. The first thing is that the categories are the labels of my model, which you see here. Now the other cool thing that we do is when you upload your data set, we basically keep 10% of the data and don't use it to train the model. Instead we use it to test how the model is performing. So in this case, on that 10% of the data we have 0.66 test accuracy score, which is actually really good given that I had less than 20 examples per category. So in addition to the test accuracy score, we also provide you with a confusion matrix, an F1 score per category, precision, recall, and a few other metrics. Now, all of these metrics are defined as well as explained how you use them to improve your model in our documentation on Einstein AI. So with that, let's go ahead and try this out. Let's get a prediction on a text and see how our model is doing. So let's go ahead. Let's imagine I just met with a student, James, and he's behind and struggling to catch up in school. So I go ahead and enter my notes and as soon as I click send, you don't even have time for a drum roll, right? It's like the model has already predicted it, right? In less than a second you see that the model has predicted that this is a tutoring referral with 0.89% confidence. Now let's try another example. Let's actually use the text that was used earlier when Iuri was showing us the college for a demo. So once again, you see here, the troubled living condition is correctly predicted. So these APIs are amazing and provide you with a lot of flexibility. But for everyone that's not a developer, for all of you admins out there, don't worry, we actually have lightning components that you can use. So you can basically go through all these programmatic steps that we just went through with clicks and not code. So we're going to go ahead and just show you that last step to not redo the same thing, but this last step here, you see, we're using the Einstein Playground, which is a lightning component for our APIs. And so we're going to go ahead and see we're using the same model ID that we just trained. We're going to go ahead and enter the same text and we're going to notice that as soon as I click send, I get the raw outputs, just like I did with the APIs, and then I also get a formatted response. Now, the great thing is that not only can you do this with qliks and not code, you can also use this to automate updating of fields like we saw updating of workflows, you know, case creation and many more. Things. Right? So to summarize, we're now generating a lot more data than we used to, and hopefully we're collecting this unstructured data, because now, thanks to Salesforce, we're able to derive value from this type of data, right? From images, text. College Forward is now able to spend less time updating tasks and triggering manually triggering workflows and more time with their students. And in your case, this means that you can help your sales reps work smarter or your service agents close cases faster or simply understand your customers better. And so, with that, back to you, Marco.

All right, thank you very much, Z. So you see, you see why I'm so excited. In the year and a half since we introduced the first Einstein product, 24 different Einstein applications have gone live. And there are so many more coming. We're moving very, very quickly. So how do you stay abreast of all of this? Well, we have the easiest trail mix to find ever, because it's the only one that has its own domain name. Trailhead.einstein.com. check it out. And there you can actually play with many of the technologies that we showed here today, even Prediction Builder, which is still in pilot. And you can also get a free trial of the programmatic APIs that Zineb just showed at Einstein AI, which also has its own domain name. And so, from all of us on the Einstein team, give it up for Katie Hurley over there in the Jeep doing all the demos. That's right. For Julian Iori Zineb, Austin Buchan, and Ashley Brown. From all of us on the Einstein team, for allowing us to join you on this trail to the fourth Industrial Revolution, bringing intelligence to your business applications, we'd like to thank you. Have a good day.