The Conference Board Webcast on AI/ML in Marketing and Beyond
In this 2018 webcast, Tim Peter (host) and Mike Moran (co‑director) discuss how AI and machine‑learning can be applied to B2B marketing, enterprise content, and broader business processes. They begin by stressing that high‑quality, well‑labeled data is the foundation for any ML effort, illustrating the importance of outcome definition with weather‑forecasting and medical‑diagnosis examples. They note that data volume requirements vary by problem – a few thousand labeled pages may be enough for taxonomy extraction, while other tasks need far more. The conversation then moves to concrete B2B use cases: automatic tagging of content (industry, type, product), sentiment analysis with rigorous coding guidelines, and improving site search by defining “successful” searches (long‑click, dwell, conversion) and using those outcomes to train ranking models. They show how these same techniques enable content recommendation, entity disambiguation, and personalized experiences (form auto‑completion, event recommendation). The panel explores the potential of AI‑driven robotic‑process‑automation discovery, noting it is theoretically possible but not yet commercialized. Chatbots are examined as a form of RPA; the speakers stress the need for a human back‑stop, hybrid rule‑plus‑ML designs, and leveraging site search as a fallback. They then broaden the scope to corporate decision‑making – financial forecasting, HR talent matching, and team‑building – all of which hinge on gathering the right data and defining success metrics. Finally, they discuss emerging roles required to operationalize AI: data scientists, ML/AI engineers, interaction analysts, and voice‑first designers, emphasizing the collaborative nature of feature engineering and semi‑supervised learning. The webcast closes with a reminder that successful AI projects start with clean data, clear objectives, and measurable outcomes.
Chapters
Hi, and welcome to today's Conference Board pop-up webcast. I'm Tim Peter, Program Director for the Digital Strategy and Enterprise Digital Transformation Councils here at the Conference Board. I'm joined here by my co-director, Mike Moran. During our discussion, you can use Adobe Connect to ask questions by typing into the chat box on the bottom of your screen. We will select a few questions for the end of our discussion, and of course, feel free to chime in throughout because we'd love to hear from you. Okay. All right, starting off, Mike, thanks so much for joining and having this discussion about AI and machine learning.
Glad to be here, Tim.
All right, so let's just start off. Obviously, AI and machine learning depends upon data, right? So can you talk a bit about the ability to leverage AI and how that's tied to having your data in a good Sure.
Yeah, I mean, I don't think there's anything more important than having your data in a kind of a clean state. And what that means for different applications can be very different. So if you're using supervised machine learning, where you have a set of training data where you're creating a model, It's really important that not only the data itself be easy to import and all in the same format, but also that you are very confident of the accuracy of your outcome data. So, for example, if you were trying to predict the weather, and the way you were doing that is you were taking all sorts of inputs of the barometric pressure and wind speed and You were trying to look at where the systems are 100 miles away and said you were using that, that type of data to predict what was going to happen in an hour or 3 hours, then in order to do that, you have to have good data that says that you agree on what it is that happened, right? So if something as amorphous as the weather, for some people, Accuracy means, you know, how many inches of snow did we get? For other people, it means, did we get precipitation at all? Because predicting snow versus rain is really difficult when it's 32 degrees out. And so what looks like a really bad miss, it says, man, we got 5 inches of snow and they said it was going to be raining. Just, it was because of 2 degrees of temperature that was wrong. And so trying to figure out what your outcome actually is, what are you really trying to predict? That's really important when you're putting your data together. The other thing that's important is that a lot of times the outcomes you're trying to predict are actually a stand-in for human judgment. So if you think about like the Watson application and how it's trying to do medical diagnoses, what's really happening here is that you have to be careful that your data is actually representative of the correct answer, not just the most popular answer, right? So we're playing Jeopardy rather than Family Feud. So, you know, if everyone thinks that's the right answer, that's only good if it actually is the right answer.
Right.
And so making sure that you understand that your data was really compiled by experts who agreed with each other rather than just anybody that you gave a survey to is a really important distinction. And so I think the Most important thing for having your data ready is that you know where it is, you know that it is accurate, and you know how to import it into the machine learning environment. So the most easily to do that is if you have data that's all in a common format. But you can bring data in that's in multiple formats as long as you really understand that the different fields in the data are all defined the same way. So it's possible to do that. So having kind of a data dictionary that says this is what it is and we absolutely know how it was compiled and it's been compiled the same is just as good as if it all came from the same source. But the real problem, as I said before, is the outcome data. You have to know what your outcome is. You have to know that it's being done in a way— that it's being tabulated in a way that is actually the same thing you're trying to predict. And you need to make sure that if it's based on human expertise, that those are really experts and that there's maybe multiple of them that agree with each other.
So that you're really training artificial intelligence instead of artificial stupidity, right?
You bet, because the thing will give you the answer you train it to give. It's just, you'd like to make sure that's a good one. I mean, a lot of the— if you think about AI the same way you think about raising kids, you know, we talk about, you know, you know, you don't want to be a bad example, right? And so why? Because the kid learns from whatever training you give them, right? So if you do stupid things, you train a kid to do stupid things, right? And it's the same thing with AI.
Like when they created that bot on Twitter, when Microsoft put the bot on Twitter and it learned how to be racist, sexist, misogynist, and all the other -ists that you don't wanna be, right?
Yeah, that's a good story because what happened is that they put something out publicly. And so instead of being trained in its behavior by experts who were trying to get a good outcome, They got trolled and a whole bunch of miscreants out there went out and said, hey, I wonder what we can get this thing to do, right? And they gave it all sorts of horrendous feedback and conversations. And so it started parroting the very things that it was fed. And so, yeah, machine learning works really well. So be careful what you teach it.
Fair enough. Fair enough. Fantastic. So just how much data really is needed to ensure that the machine learning algorithms are learning?
Well, I mean, that's again a good analogy as a kid, right? If the kid doesn't know anything, it learns something right away. You know, I mean, the difference between zero information and a little information is a lot. But maybe it didn't learn to be an expert right away. And so if you've, if you've already given it enormous amounts of data and it still isn't quite as accurate as you would like it to be, you might have to give it way more data in order for it to really get that little bit better.
Right.
So it's not really kind of a generic answer to the question. And the other thing that's true a lot of the times, because a lot of times when we work with companies to try to help them with these problems, they'll say, they'll say things that are not quite the same as your question, but they'll say things like, well, how long is it going to take?
Right.
Before this works? And the truth is, we never know. We've, we've done projects where it Took a couple of months. We've done projects where it took considerably longer than that. And the reason we don't know is not because we don't know if the algorithms, the machine learning systems, are going to work. It's because there's 2 really big unknowns when you do any— this project. One is you don't know how good the data is.
Yeah.
So going back to the first question you asked, the data might be garbage. And if it's garbage, then it's gonna be pretty hard to get signal out of the noise. But the other thing that we don't know is how good does it have to be? We did a project once where in just a couple of months, we were able to get a 75% accuracy. Now, for a lot of projects, that's not really good enough. A lot of projects, they need a lot better than that. I mean, obviously, Watson's diagnosing your medical condition, I don't think you'd say, yeah, you know, 3 out of 4 is fine, right? I mean, But in this case, the current system they had had 15% accuracy.
Right.
We were actually 5 times better. So even though this was kind of our first swing at the pitch, and, you know, it only took us a couple of months because it takes a month to just get the data together and then another month to start playing around with it, they were ecstatic after just— after that short a time because it was so much better than what they had, right? So a lot of it is, you know, what are you comparing it to and what's good enough? And a lot of that depends a lot more on the problem and on your data than it does on any of the techniques that a specialist could throw at this.
You know, I know you've told this story before, but can you remind us the one where it had 9% accuracy or something like that? Wasn't that the—
Well, there was one, yeah, there was one where we did a personalization application in which we were putting ads on the screen that were actually tied to the person's industry.
Hmm.
Showing them industry-related content, and it got a 9% click-through rate. And so you might say, well, you know, 9% click-through rate doesn't sound all that good, but the generic stuff that we were putting up before had a 1% click-through rate. So it was 9 times better than what they had. And so even though 9% maybe doesn't sound that great, it was really good compared to what they had in that space before. And so You know, as you say, that's the kind of thing that it's more about the problem and where you're starting from and what kind of data you have and what's good enough than it is about whether the techniques work or not.
Right, right. Is there a, you know, is there a case where there's too little data, where like you just don't have enough data?
Yeah, there's plenty of times that that's true. There's, we've done a couple of projects recently where we're doing, We're extracting topics or subjects, themes, whatever word you want to use, like whatever it's about. You know, we're using web pages and looking and kind of giving them a taxonomy of subjects. And what we know is that if you have fewer than 1,000 pages, it's— there's just not enough for it to really work that well. There's not enough different examples of 20 or 30 or 50 different topics. that you really get good results. But once you get over that threshold, then it's good. And I mean, obviously, if we're working with large clients that have tens of thousands or hundreds of thousands of pages, it works really, really well. And so it depends on the problem, how much data you need. But yes, there are definitely situations in which it wouldn't make sense to use machine learning because you just don't have enough data.
Makes sense. So let's take that a little further. You know, are there any general B2B marketing use cases that lend themselves to AI better than others?
I think that that kind of automatic tagging is one that works really well in enterprises because you typically have enough data that you can automatically figure out the industry, you can automatically figure out the topic, you might be able to automatically find things like what kind of content type it is. So is it a white paper? Is it a case study? Is it a product spec? There's a lot of other kinds of auto-tagging that can really be helpful as well. And so I think that auto-tagging function is really useful. We've gotten— we've had success using it for site search where we use feedback mechanisms to actually figure out which searches are the most successful. So then we'd like to suggest those searches in your autocomplete when you're typing things in on the search bar. Also other things in site search that you can use this for. Another thing that works really well is sentiment analysis. I've worked on systems where if you just try to look at the words and just try and figure out whether the words say that they like something or don't like something, you get very little coverage of the data, but if you actually use experts to take a look at the tweets, for example, and help them figure out which ones are positive and negative, when you amass enough examples, you can get some really good outcomes.
Sure, that makes a lot of sense. Do you mind sharing the story about, you know, knowing whether something is positive or negative and how you, you know, how you had to train that case specifically?
Sure. Well, yeah, one of the things we found out when we first started with that was that only about 89% of the time do humans agree with each other. It's kind of a hard problem, right?
Right, right.
I mean, so if you took just one person's opinion of all the— in all the training data, you could never get past 89%. You would probably— you'd have to be lower than that because the training data itself is only 89% accurate, right? So, yeah, so what we did is we kind of tightened up what they call the coding guidelines, right? So if you have a human being, you can think of what the human being is doing is coding the outcome, right? So in this case, they're coding, is it neutral, is it positive, is it negative? And so—
Like when you're coding a survey.
Correct. That's where the name comes from.
Yeah.
Think of like putting together guidelines for whatever it is that they have to do so that you know that things are being captured accurately. So one of the things we did was we had to code based on who the point of view was from. Because like if you think of a tweet that says, you know, I'm so glad I'm with Verizon because my friend's Sprint phone drops the call every time she's in the car. Is that positive or negative? Well, it depends on whether you're Verizon or Sprint.
Right.
It was something that was very unclear when we started out with the coding guidelines. You just said, is this positive or negative? And people— and we found all sorts of variations. Some people thought that was positive, or people thought that was negative. And the truth is, if you say, okay, if you're Sprint, is it positive or negative? Then everybody agreed it was negative. And so a lot of the times, just trying to put together what the rules are to say how you decide things is really important. We had other clients that if someone shared a link but they didn't say anything positive, they wanted that to be neutral. Other people said, no, if they're sharing our content, that's positive all by itself. And so they wanted that to be positive, right? And so different people can have different opinions as to whether the same act is positive or negative. And so those are the kinds of things that in the coding guidelines you just Got it.
Makes sense. So, you talked about site search a moment ago. You know, how is AI or machine learning being used to drive more accurate and more relevant site search results, right? Are you using query history of the community, the individual? Are you doing suggestions, personalization? I mean, can you talk about that a little bit? Yeah.
So, we mentioned a few minutes ago about a way of using suggestions, right? So that, You don't have to just suggest what's popular. You can suggest what's popular and itself successful, right?
Mm-hmm.
And that can buy you a few points of success right there as well. Now, in order to do that, you have to decide, well, what makes a success? How do we measure a success? So what we tend to use is that the keyword gets results, it gets a click, and the person stays on that page. They don't bounce back.
Right.
We're looking at some more capabilities where we can actually tell whether someone engaged on that page. Did they scroll the page? Did they click on something? And all of those things kind of go into your idea of whether something's a success or not. And so by using that, now you have outcome data that is somewhat objective. And so you can use those outcomes to then start to tune things. So what you can do in the algorithm is you can look in your ranking algorithm and say, okay, in addition to all the things that ranking algorithms usually have it, right? They're looking for the words on the page. They're looking for all sorts of things, right? We're gonna actually say we want to boost content that itself has been successful when someone searched for this keyword.
Right.
And so by taking those analytics, that's the data that you have to have in good shape. By having that data in good shape, Then you can put that into the algorithm, and then now you're using AI capabilities to actually improve your search ranking. There's other things that you can do in site search. The auto-tagging work that we talked about is really important. There's also work that you can do in entity analysis, where if someone enters something, they— suppose you had a database of companies in it. And someone entered Sprint. Well, Sprint could mean a lot of different things, right? Sprint could be the company. It could be a NASCAR sprint car race. It could be the high school running race. It could be, you know—
An agile development process.
Exactly. There's a lot of meanings for it, right? So if you actually use text analytics to identify what the meaning was of Sprint, then when someone enters Sprint into that database looking for companies, you can be pretty sure they're looking only for the versions that are talking about companies. And so then what you can do is you can say, okay, let's only find those, right? So between looking at the text analytics on the backend where you're indexing the data and looking at the runtime analytics on the frontend where you're identifying which things people are searching for and you have an idea of which things are successful or not, there's many ways of applying AI to site search that can dramatically improve the business results. And that's really what you're worried about is because It's not enough to just say, hey, you know, we think the search is working well. What you're really trying to do is to say, look, there's business reasons why we have search.
Right.
Either trying to get people to buy more from us, right? And so we should be able to show how successful searches are tied to additional revenue. And if you have a good attribution model, you can usually do that. You can also kind of figure out how you can avoid costs. So there's a lot of clients that we work with where there's where they're doing it for support reasons. So they're trying to answer questions on the web so that people don't call them up and spend $50 or $100 on a phone call. And so there's ways of measuring whether you avoided calls as well. And so both of those things are things that you tie in as part of your data. Now, if you have those kinds of attribution models where you can say this did avoid a call, or this actually led to, you know, an additional conversion or a market-qualified lead, then you can use that as further outcome data to say that your search was working. And so now you can tune it even tighter. And so depending on how deeply you can get into the analytics, um, some things you can really turn into a great machine learning model.
That's great. That's fantastic. And I know you already answered this a little bit, but somebody asked, and I just want to, I just want to come back around to this just for Clarification. So, you know, somebody asked, you know, so we're defining searches how, successful searches how, and it really starts with what Google calls the long click, right? They leave the search results and they don't come back is where we start, and then we build from there to make it more sophisticated over time.
Correct. Yeah, what Google calls the long click means that, first of all, you got search results, you clicked on something, and you stayed there. Right? You didn't bounce back to the search results page. And so that's the simplest definition. It's very easy to measure that if you have the right tools. But we're actually looking at deeper tools that actually look at whether you scrolled the page, did you click on something on the page that the search result took you to, right?
Mm-hmm.
So you go deeper. You can also tie in whether they actually completed a conversion or completed a task. There's lots of ways of measuring whether a search is successful. And so depending on your situation, you know, you would try and look at which things really make the most sense to define as a search, a successful search.
Mm-hmm.
The reason it's important to define that properly is the same reason we talked about coding guidelines before. You want to make sure that you're identifying what you legitimately think is a successful search. Because if you're doing that, then you can start tuning the machine learning model to tell you how to optimize for that outcome. So if you have the outcome defined properly, now you're really going to be able to move the ball by using machine learning against it.
So this is a great transition. You know, somebody asked what kinds— and I'm going to interpret this a little bit. The question was, what kinds of AI or machine learning drives more personalized digital experiences? And I might change that a little bit to say, you know, How are people using it to do that, right?
So, so this— so when people talk about personalization, there's usually a bunch of different things that they mean by that. Yeah, something as simple as, you know, hi Tim, when you come to the website.
Right, right, right.
AI for that, right? There's— there are other types of applications where you can use some AI in it. So things like form completion, sometimes you might be able to recognize a person, be able to So, you know, you might have a form that says, you know, please fill in the right items on a form. You might initially say, well, how does that need any AI at all? Well, you would be surprised at the different variations that you might have in your enterprise over what the name of the field is on the form. Sometimes it might say login, sometimes it says user ID, sometimes it says email address. They may actually all be email address.
Right, right.
So, and so if it has different names, you have to use some type of AI to try and figure out this is what we think the right answer is.
Right.
And there's often situations where it's even a lot harder than what I just said, right? So, so for form completion, you might have a little bit of AI. The place where you use the most, in my experience, is what we would call content recommendation. So it's similar to the thing we talked about earlier with the, you know, showing ads based on their industry. That's kind of a content recommendation personalization application. So it's— what it's really doing is it's looking at all the content you have and and deciding which content to show. Now, marketing automation systems in the past have been the only way to do this, but they use— they do it using rules.
Mm-hmm.
So they would say, if this, then that, and those things work, but they are fairly expensive to put together and they're fairly brittle, because if you can imagine, it says, if this, then show that. Suppose what you wanted to show was an event. You say, okay, if this person's interested in this type of product area, then show them the event coming up that's in their area for that product. And it's like, well, that seems like such a great rule, but what it ends up being is it ends up being, show them this URL. That's what the hard-coded rule is going to say. And what happens a week after the event? Did anybody really remember to go back and fix the rule? Usually not.
Right.
Usually you have to notice it's broken before you go fix it. And so if you use AI techniques, those kinds of things can be handled a lot better, because you can put a rule in that says, yes, I want an event, but I want to make sure it's an event that's happening in the future, and that it's an event that's in this person's area, and that it's an event that it's about the right topic. And so what— and most of the time when you use content recommendation, the way that we tend to recommend those applications work is with a site search engine. So if you can make your site search engine smart enough that it actually knows which pages on the sites are events, it knows the location of the event because we did some of that auto-tagging where we said, hey, we figured out where this event is, let's tag that and make sure that we're only showing those types of events. Then you can use site search to do things based on the subjects like we talked about earlier. Is this a subject they're interested in?
Mm-hmm.
You can say, is this near me? Well, how would you do that? Well, a lot of site search engines have geolocation capabilities. So if it knows the place where the event is and it knows the place where the person is, it can actually show that if it's within a certain number of miles that you set up as a rule. Oh, yeah. Um, and then it— and obviously it can also understand what the date of the event is using that type of auto-tagging capabilities as well. And in your rule, you just say make sure that the date is you know, after today or after next week or whatever rule you want it to be. And those rules will always work. Now, if there's no event that fits, then it'll put something else there. Right. You know, and so you can have several rules in there and you can control what's in the rules, but you don't have to hardcode it to content that's constantly changing. You can let the search engine be your surrogate for finding the right content. Now, because it's a search engine, it can actually make sure that it's showing things that are the most relevant, right?
The way—
but it can also be showing things that are the most popular. So using all the same kinds of tricks we talked about before for site search to say which are the things that people seem to stick on, it can actually make sure it's showing those pages too. And so your ranking algorithm can be very rich, and now you've got a really cool way of doing content recommendation. And it's all based on just making your site search engine better, which is technology you already have and you want to make work anyway.
And to your point, it's not brittle, right? Because somebody can't come up with the use case that finds the gap in the rules, right? Because the AI knows enough to say, wait a second, I know how to answer this, even though there's no rule there, right?
That's exactly right. So instead of hard coding things that people have to remember to maintain, You set up a system that's robust enough so that as the content changes, then the system automatically changes too. That's great.
That's great. So we've got another question from our viewers who— somebody asks, we've begun to use robotic process automation. You know, could new automations be automatically discovered and created using either AI or ML?
And the answer is theoretically yes. You know, for those that— But I haven't seen it done yet, right? So for people who don't know what robot process automation is, it's basically you can think of it the same way you think of testing user interfaces. So if somebody's testing a user interface, what they're doing is they're trying to have kind of what's the desired input and the desired output. And so with robot process automation, it uses a lot of the same kind of technology to automatically fill out a form. Form or automatically conduct some task that you might have a person doing by rote right now. And so if you can get enough examples of people correctly taking the input and creating the output, then you can use robot process automation to do that. And so some of the things that you could do to solve that kind of problem, then— and again, I can imagine how it could be done, but I actually haven't seen anybody go out and do that yet. But what I can imagine you doing is going out and looking for opportunities where you've identified how to recognize that this is something that's used a lot, right? So you can imagine using— if it's a web app, you can imagine using web analytics that says, okay, we're only going to look at the pages that are used, you know, thousands of times.
Mm-hmm.
Because those are the good candidates, and we're only going to look at the ones in which people are entering things, right? Because if they're not entering anything, then it's not, it's not a good candidate, right? And we're going to look at these things internally, and we're going to actually look at which ones seem to be entered by the same group of people all the time. Maybe they're all entered by one person, maybe they're all entered by 10 people, but we're not looking at ones that are entered by 1,000 people because that would— that's probably where you're collecting data from a wide variety of people. It's probably not a process that's going over and over again. I can imagine the kinds of things you'd look for in an AI application to identify this stuff, but I actually don't know of anybody who's doing it yet. So looking for those kinds of opportunities, I think, is something that is possible, but it would probably be— you'd probably have to hire a consultant who would come in and help you figure out how to do that, who has this kind of capability, because I don't think there's anything off the shelf doing it. There's a lot of off-the-shop RPA capabilities that can help you to automate specific situations that you have, right? A lot of them use AI, but in terms of looking for opportunities, I haven't seen that yet.
Got it, got it. Perfect. All right, so shifting gears a little bit. So obviously one of the things people talk about a lot with AI and ML and things along those lines are chatbots. So can you talk a little bit about, you know, the relationship between AI and chatbots and how chatbots improve customer service? I mean, is this something people should be looking at?
You know, I'm hard-pressed to tell people not to look at stuff, right? I would always look at whatever it is, right? Because you never know if it could be something that's useful to you. For chatbots, I think one of the things that you really want to focus on is really the same kinds of things we just talked about. We're talking about how would you identify an opportunity for RPA, Well, you can think of a chatbot as being an example of RPA, of robot process automation, because what you're really doing is you're taking things that people do and you're trying to automate them. And so chatbot's kind of a good example of that. And so if you look at, for example, a place on your website where customers are coming in and opening up chat windows and asking questions, You can say, hey, if we have areas like that where lots and lots of questions are being asked, well, yeah, that seems like a good place to try out a chatbot. Now, here are the kinds of things to think about when you're looking at a chatbot for that kind of situation. The first thing to think about is if you think you're going to employ a chatbot, but you don't think you're going to have any backstop for that, It's not going to work.
Right.
There's no way to kind of say, hey, you know, we don't offer chat on our site, but I'd really like to offer chat, so let me just get a chatbot. Now we can say we have chat on our site. It's like, it just is not going to work because there's too many variations to the questions. It's too easy to stump the chatbot.
Right.
And then you end up with maybe a worse customer experience than you had already. Now, if on the other hand, you already have a team that's doing chat, Then you're probably in a lot better shape to do this because then what you're trying to do is you're trying to use the chatbot to automate the early parts of the conversation, automate simple conversations. So you can think of it as the web version of the phone tree for the call center. Now, luckily, people seem to like it a lot better than they like call trees. They really feel like they're getting close to their answer with a call tree. They're like, okay, stop talking so I can press 5, right? So, but the chatbots are easier because the interaction that they have is pretty much the same as the interaction with a person when it's working well. And so then what you're down to is trying to understand, okay, so what happens when a chatbot is in action? How do we use one? How do we pick one? How do we train it? And the truth is that there are many different kinds of chatbots, and the answer to that question is pretty much all over the map. There's a couple of things that you want to think about. One is whether you've got all sorts of records of your existing chat that can help train the chatbot. There's 2 ways the chatbots get trained. There's kind of a rules-based way, which is the way most of them unfortunately work today. And I say unfortunately because they have all the same problems that we talked about before of being brittle, And having people know how to change the rules. And so those are really difficult to keep working.
Right. I think what you said is, you know, the way they mostly work today.
Yeah, I think I did say that. So that's the thing. So that's really hard is that if they're only based on rules, then you're going to find that it's, it's actually fairly difficult to not only have a good experience, but to keep them maintained.
Right.
Working right. Most of the really good ones are using a combination of rules and machine learning. So what they do is they are— if you've got records of all the chats you've ever done since year 3, then you can take those out and you can, you can start putting them into the system and it starts to learn what things people ask and it starts to learn examples of correct answers. And so those are— that's a much better way for a chatbot system to work. Now, having said that, I will tell you that it is utterly impossible to have a chatbot answer all the questions that a human being could answer. And if you think about it, maybe a chatbot could do what most companies would call level 1 or even level 2. They're never going to be able to do the more difficult problems. Right? And so, and so then the question you have is, what is it that you're using as your backstop? You can— the easiest thing to do is to use your people as the backstop. And so instead of thinking of the chatbot system as being a way that you're going to fire your call center, what you really need to be thinking about is how you're going to use your current call center to scale bigger, because they'll be able to handle more chats than they do today because the chatbot will be answering a number of the questions for them. Now, the other thing that we've seen with clients that can work really well is you can actually— and I'll beat this horse again because people have asked about it— you can use your site search to be a backstop for the chatbot. If it asks a question that the chatbot doesn't know the answer to, If you've done a good job tuning your site search, you might be able to come up with a few different possibilities and have the chatbot execute a search, get back 5 things, and say, are any— do any of these things seem to be what you're asking about? Right? And even if they don't actually get the answer to the question from that, just them picking which one of them could be the answer can give you more information. And it also gives you more time where the person might answer their own question before you have to hand it off to a human being. And so, and that would really give you a lot more coverage than you can get from training the chatbot system, because you can imagine that a lot of these questions are one-offs. I mean, if you use site search data, it's very common for only 50% of the keywords to be something that are entered more than once in a month, right? 50% of them are only entered once, right? And so Think about that when it comes to asking questions. A lot of these questions are questions that no one else is going to have for a long time, and so if you can send them to the search system and take a shot at it before giving up and throwing it to the human being, that can, that can also kind of lower your costs and scale your existing personnel to more user interactions.
And actually, there's a bunch of real-world examples of that today. I mean, if you look at how Kayak and Expedia handle their chatbot through Facebook Messenger and places like that. That's exactly what they're doing, right? They just bring you to a search result at a certain point.
Right. And that can work really well if your site search is working.
Right, right.
That's the thing that this goes back to. That's the thing that the content recommendation goes back to, is that this is kind of your secret weapon in finding answers for people, finding content for people if you've really tuned it well. Then it can— you'd be amazed at how well it can find things and do that automatically rather than relying on people.
So, while we're talking about content, and this is really a lot of our discussion of content up to this point has been more on the customer-facing side, but, you know, how can you use AI or ML to improve the quality, the normalization, the relationships in product information or other, you know, other content that's back of house? Can you talk about that a little bit?
Yeah, there's a lot of it is using those auto-tagging technologies. So you can do relationships between different types of information because you can identify that they're talking about the same topic. You can do entity analysis that says, I can identify product names. And so I can see that all these pages are talking about the product. Some of them are using the product name a lot. That probably means that this is a page about the product. Other pages are referring to the product name once. That probably means that it might be the opportunity for an upsell or a cross-sell because it's related, but it's not really about the product. It's about some other product. And so there's ways of using that kind of entity extraction and auto-tagging to be able to do a lot of that kind of relationship work. Now understand, it's probably not going to do a perfect job, but what you can do is you can use what they call semi-supervised machine learning, where you put a human in the loop. And what they're doing is they're— the AI is actually suggesting the relationships, and the human being is making the final decisions. Rather than the human being having to go out and ferret out all of those relationships everywhere, right?
Yeah.
Which would be very, very hard. The computer is the one that's actually looking for everything, and then it's showing some things and says, these are 10 that I found, how many of these are good? And what happens is over time, it starts to develop more confidence about certain types of relationships. And you can sometimes say, okay, hey, look, if the machine is 85% confident, then we're just going to tell it, put that relationship in.
Yeah.
And if it's under 85%, then refer it to a person. And by the way, if it's under 50%, it's just wrong. Don't show it to anybody. Just forget those, right?
Right.
And so after you use this for a while, you start to develop those kinds of thresholds in the confidence level. And now you have a system that can really start to find the relationships and keep them up to date. Because as the content changes, every time it changes, it can look again, right? Because if somebody changes that product information and it's no longer referring to the other product, then maybe that relationship is broken. And so now you want to get rid of it, right? And so that's how this stuff can really help you maintain those product relationships.
That's great. That's fantastic. So, okay, up till now we've talked a lot about content, we've talked a lot about process, we've talked about all those kinds of things. Let's talk more on the people side. You know, are there new roles or skills that are needed to really take advantage of AI and machine learning?
You bet. So, I mean, I mean, I don't think anybody who's interested enough to tune into a webcast on AI would not have heard of a data scientist, right? So, I mean, that's obviously a relatively new role, and it's a critical one because the data scientists are actually the ones who are analyzing the data, who are trying to figure out what in machine learning they call the features, which is what the aspects are of the data that are kind of the characteristics for the system to look at to do its pattern recognition. So the system can't just look at all the data and do the pattern recognition. You know, someday it will be able to do that, but today that really requires too much compute power. And so it's really important for experts that are both data experts and also experts in the problem to do that feature analysis where there's— where, for example, when we talk about content and we're talking about what that relationship is between the products, I kind of gave you kind of this— not exactly a rule, but it's kind of this thing that says, hey, if it uses a product name a lot, maybe it's about that. If it uses a product name once, maybe it's related. The machine is not going to figure that out all by itself. You need an expert to kind of suggest that as a pattern, and then the machine might be able to say, okay, yeah, I can see this, this says the product name 9 times, so it's probably about that. This uses the product name once, so it's probably related. Um, this one uses it twice. This one uses it 4 times.
Right.
What do you think of that?
Right?
And so it can kind of find other cases that you hadn't really thought about and say, hey, look at this one. And it could be that of the ones that use it 4 times, some of them are about the product and some of them are not. And so they say, hey, look, the ones that are using it 4 and 5 times, they— some of them are about the product, but let me show you which ones. These are about the product, and these ones not really. They just happen to use the name a few times. And so, and so it's then going to look for other patterns to try and figure out how to distinguish those things. And so data scientists are the ones who can help do that feature analysis along with the experts. Another role that's starting to emerge Is— they have different names for this. Some of them calling them machine learning engineers, some of them are calling them AI engineers. They're not data scientists, but they are technical people. Often they're former programmers who know how to operate the systems. They know how to operate a training system. They know how to operate a prediction system that's running in real time. They know how to wrangle data.
Mm-hmm.
So they, so they can do a lot of things that you wouldn't want to pay the data scientists to do anymore, because the data scientist's making $300,000 because last week Google called them. And so now you're trying to figure out, okay, how can we give helpers to the data scientists who don't make quite that much money, so that we're using the few data scientists we're able to keep here that have not gone to Silicon Valley? How do we keep them here?
Right.
and leverage them for as many problems as we can get them to solve, right? So those types of roles are emerging as well. Um, we talked earlier about chatbots. There's a role that's emerging for chatbots and for a few other types of AI things called an interaction analyst, Mm-hmm. where it's not really the same as do— as a, as a design person, Mm-hmm. who would design a webpage, or even a user experience person who may understand the cognitive model for a problem. There's a very specific thing that says, we know how to get the chatbot to ask a question in such a way that it gives us a good answer that we then go off and do something else with, right? So there's kind of an interaction person who's starting to emerge as a specialist, especially in chatbots, but it's also starting to come up with voice-first designs.
Mm-hmm.
So if any of you have an Alexa or a Google Home, or you use Siri on your phone, there's a lot of thought that goes into exactly how those systems speak to you so that they get an answer that they can understand and do something intelligent with. And so those types of interaction designers is a new specialty that's emerging too.
That's great. That's fantastic. So we've got a couple last questions and we've got just a handful of minutes left, so I'm just gonna Head through these, but I also want to remind folks, if you have any questions we haven't asked so far and you want to pop them into the chat box, rather, excuse me, we'll try to make sure we get those in before we run out of time.
Sorry, real person answering questions.
Exactly. Unfortunately, real person who can't read them. So, how can artificial intelligence or machine learning be used to improve corporate decision-making? You know, could you do it? So that you reduce the number of bad decisions made, you know, things like mergers and acquisitions or, you know, things like that, HR management, etc.?
You can definitely do that. I mean, if you think about a lot of the financial decisions that companies make, some of the most money that's been invested in AI over the last 10 years has been done by hedge funds and other types of folks who are actually analyzing stock prices on Wall Street. They're reading every piece of data about companies, right? They're not just scouring their EDGAR filing. They're going in and looking at every news story that's out there on the company. There are— there's a project that I worked on that actually uses sentiment analysis in social media to actually predict ups and downs in demand tracking.
Mm-hmm.
I mean, you can imagine All the kinds of things that are going on. So those kinds of financial decisions, I think, yeah, there's been a lot of work done there. In terms of HR decisions, I think we're starting to see more of that. I think that there's been an awakening the last few years in HR where people are starting to understand that who a person is and what their strengths are is really important. I know we have somebody on the council from Gallup who does StrengthsFinder.
Mm-hmm.
And so there's a big focus that people have been doing on gathering data to look at, you know, how you describe a person. You can also use that same kind of technique to describe what kind of situation they're in. So if there are certain people that do a little better under stress than other people— I know a lot of people like to say that they're great, they're better under stress. I think we know nobody is actually better under stress. There's some people who are less affected than others. And if you could judge what the stressfulness of a job is, then by using those types of things, you might be able to figure out who are the people who might do better at that or worse than that. Um, you might find that people work differently with different kinds of people. So maybe being paired with a different kind of boss might be useful, right? And so all— and so the first thing you have to do, like we talked about in the very opening question, is get your data, right? So a lot of— in a lot of cases, we don't have data that really describes people.
Right.
We don't have data that objectively describes the situation they're in. And so gathering that data is the most important thing. If you can collect data about it, and then you can start to see what the outcomes are, right? Where outcomes are, can you really decide whether a person succeeded or failed in a situation, right? That's very binary. You know, can you really say that it was based on the person, or was it the situation that was going to fail just about no matter who was there, right? So I think collecting the data here is going to be the hard part, and if you have that data, now you can start to use these types of techniques to make better decisions about personnel, for example.
Got it, perfect. And there's one last question, which is very related to what you were just talking about, but just if there's any nuance you would add to it, you know, can you talk about AI or machine learning being used to improve working relationships, you know, perhaps by doing a better job at assembling teams with the right mix of skills, temperaments, things like that?
Yeah, I think it, I think it's mostly the same kinds of things I was talking about, right? Is that first of all, when you say you have to define what success for the team, right? Right. If success for the team means that they performed really well and you have metrics around that, is it that they liked each other, right? They stayed together for a long time, you had lower turnover. I mean, what, what means that a team was successful? Right? That's one of the first things you would have to tackle there. And the second thing is the same thing we talked about before. How do you get enough data points around people that you can then start to see how these things work, right? So, there's been a lot of work that's been done recently around analyzing email. And so, can you tell from email whether people are working together well, right? That would be a really rich source of data. Now, is it true that if people are working together well in email that they're working together well in every other way?
Right.
Some people would argue yes, because email is actually a very difficult communication mechanism. And so if they're doing well there, they're probably doing well everywhere else, right? And so, but maybe that's not true. Maybe you're just going to find out which people are very polite and they're not going to trigger an email, but they're actually miserable and hate this person.
Right, or super passive-aggressive in a way that doesn't show up in the data, right?
Exactly. So again, all of these things are possible, right? The techniques are out there to do this, but you have to know what your outcomes are, and you have to know that you have enough data for it, and you have to have enough features, right? That's what all this input data is that you trust in order to be able to build up enough that you can actually get answers there.
And those are probably a great place, you know, to start. I mean, as we're wrapping up, I mean, if you think about it, you know, do you have the data? Do you have your arms around the data? Do you know what your objectives are for this? You know, if you can actually pull those things together, you're sort of setting yourself up to go in the right direction from the outset. Is that correct, or anything you'd add to that?
It is, and if you think about it, it's not different than if you were setting a person off to do an analysis of something, right? If you told a person, hey, I want you to analyze our competition, tell me which companies we have to worry about, what are the last 3 things they did that I should know, then you— first of all, what— how are you going to do that? Well, you have to define what your outcome is, right?
Right.
What's the outcome you're looking for, right? Can you define it so clearly that the person knows when they've done it properly? So if you can do that, now all of a sudden, then you have to open up that person to all sorts of data. Where is she going to look for this information? Is this— is it obvious that she can just go out and Google it? Or is it something where there might be some— and I think you're going to be able to do exactly what you're looking for. You'll be able to do it. Otherwise, I think you're in a situation where a person wouldn't be able to do it very well, and you can be pretty sure that the AI won't either.
That sounds like a great place to wrap up. Out of questions and we are out of time. So I'd like to wrap this up and say thanks again to you, Mike. I really appreciate it. Thanks to the Conference Board for hosting. And thank you, the folks who have listened in today, for your participation and your questions. With that, I hope you all have a great day and a great rest of the week. Take care, everyone.
