Keyword Knowledge Graphs: Using AI to Create Outside-In Taxonomies

Hi everyone, welcome to today's Biznology digital marketing webinar. I'm Mike Moran, founder of Biznology and senior strategist at Converzion, Reveal Context, and Solo Segment. I'm the co-author of Search Engine Marketing Incorporated and Outside-In Marketing and the sole author of Do It Wrong Quickly. I'm a veteran of IBM, managing groups on IBM.com for 8 years and retiring from IBM in 2008 as a distinguished engineer. We have an exciting webinar today because I'm joined by James Mathewson, my co-author of Outside-In Marketing, who's going to be giving us some insights into using keyword knowledge graphs I'll be asking James questions throughout the next half hour, and we encourage you to suggest questions for me to ask through the Q&A tab. You can also chat with other attendees through the chat tab, and you can tweet about the webinar with our hashtag, #BizKnowWebinar. We can't get started without giving a shout-out to our sponsors. Geris Corp is a full-service digital strategy firm that reaches deeper into the conversation than any other agency anywhere. And SoloSegment— revenue is trapped in your site search. SoloSegment's Site Search Inspector can help you set that revenue free. Let's dive right in. James, you have a great deal of expertise in SEO and keyword knowledge graphs. For those who don't know, James is IBM's distinguished technical marketer for search with 20 years of experience in web editorial content strategy and SEO for large and small companies. James, what else should the audience know about your background?

Well, I guess primarily I'm an author. You mentioned Outside-In Marketing, the book that we co-wrote. I also wrote a book called Audience Relevance in Search and about 1,700 articles, including a monthly article on Biznology website.

Terrific, James. Thanks so much. We're so glad you're here today. The first question I have, just, just to get everybody up to speed, is What is a knowledge graph and why is it important?

A knowledge graph is a set of relationships between things, entities. And in the case of, let's just say, a product knowledge graph would be all of your products and their relationships, most of which are hierarchical. We have, you know, a product might be related to a product family, which is related to a brand, which is related to a business unit. And then there are also what we call polyhierarchical relationships where Let's just say you have a family of products that is related to another family. So for example, Watson Analytics runs on the cloud, so you might say that there's a cloud component to it and a Watson Analytics component to it. And then you also probably need some services to help people migrate or integrate things. So there's cloud services, and those would be 3 products or services that are related to each other horizontally, as it were, as opposed to vertically.

Right. So I'm getting the feeling that it's kind of, it's kind of what people might normally call a taxonomy or an ontology. Is that, is that kind of the right idea? And maybe you could expand on that a little bit to kind of tie in how a knowledge graph compares to a taxonomy or an ontology?

Yeah, an ontology is, is, it's an ontology, but a knowledge graph is sort of the visualization of it. You can see it The nodes in a knowledge graph, like one of these giant maps with all these things connected to each other. And a taxonomy is, is a kind of, I guess, a kind of ontology in the sense that it's, it's a set of relationships, but all of those relationships between these entities, these things, are hierarchical. In other words, they fit into silos or buckets, whereas an ontology is what we call polyhierarchical. These things, you can define relationships that go between them. And some things can fit into multiple categories and things like that.

Whereas the taxonomy, each item is only in one place and can only have one relationship up and down, right?

Correct. Correct. Okay.

So for, for those of us who aren't kind of library science majors and we don't really, you know, think about it, what's the benefits of using an ontology as opposed to a taxonomy in kind of business terms?

Well, the biggest benefit is that it's, it becomes very difficult to manage data if they can all, if it can only sit in one bucket. And examples where we have these problems like cloud security, is that a security topic or is it a cloud topic? And if you have a topic that actually encompasses both, then you can help the teams that build content around it or actually products and services, you can help them understand how they can work together to solve The customer problem. Customers don't really care where you bucket things. They really just care that you can solve their problems. And an ontology is just a lot more flexible in helping them do that.

I get it. So it can feel kind of constraining and almost artificial to have the requirement that things can be only in one place. That makes sense. So let's get specific about how you've used keywords in your knowledge graph. Tell us where the keywords are from and what the purpose of them is and what is it that keywords in a knowledge graph do for you?

Well, so the main thing— I'll start with the business problem that we had. The business problem is that for a lot of the keywords that we were finding in our keyword research tools, we had no content related to them. So there were keywords that we thought our target audiences cared about, they expressed questions or problems that our target audiences needed us to solve, but we didn't have content on it because it didn't seem relevant to the products that we were trying to sell. Most of the marketing that we do starts with a product and you try to build up from there. You know, maybe you can think of it back in the client journey starting with like somebody purchases a product and working backwards, you get to a point where they're not asking about the brands anymore. They're asking about You know, how do I migrate to the cloud, for example? And we might have several brands that ultimately would help them do that, or maybe a solution that combines several brands. So how do you know that these keywords that people are searching for are related to the brands? The brands are the ones that have to create the content, and if they don't create the content, then we have these huge gaps. That's one type of problem that we had where, you know, we just could not— we couldn't build the content because we couldn't convince the business units that it was relevant to their to their target audiences. The second kind of problem that we had was that we oftentimes, for very popular words, multiple brands and business units would want to build content around it because these are things that are more obviously related to their target audiences and are high volume. In other words, there's a lot of queries in Google. So, you know, they could get a lot of organic search traffic and engagement from them. Well, we ended up with 40, 50 pages related to the same keywords. And that just gums up the works and doesn't help anything. Google would, you know, do things like de-indexing and things to try to maintain, you know, so that they could figure out what basically our mess. And we wanted to, you know, reduce that mess by basically governing keywords by the business unit that is ultimately going to be building the content around it. So we thought, well, the best way to do that is to associate products to keywords, and then the products roll up to the business unit, so then they can Then that way we could govern it. And if you want, I can describe how that works, but I don't know if you—

Sure.

Oh, okay.

Yeah, well, I mean, you know, let me try and summarize so I make sure I'm following. It sounds like what you needed to do is you needed to persuade the business units that you needed content in areas that they might not have thought, and one of the ways you did that was by showing them that there were an awful lot of Google searches about it. And then but you had to kind of make sure that you were showing them keywords that were related to their products. So that's when you created the ontology that said, here are the keywords that relate to the products. And the reason it had to be an ontology and not a taxonomy is 'cause sometimes keywords related to multiple products. Did I follow that?

That's exactly right. That's exactly right. Yeah. And so, and right. And then it's a question of probabilities, like to what extent a, keyword relates to a product. It might relate to several products, but closer to one than another. And so then you can kind of build these models that help you map keywords to products and product families. And then you can govern those keywords by the product families that are most closely aligned with them so that then you can say, like in our keyword, our SEO management tool, which is called BrightEdge, We could bucket the keywords by business unit, and then the teams that build content for those business units, they know what their keywords are, and then they can start to prioritize the content efforts based on the keywords that are assigned to them. And it makes it much more manageable. And then you can track and, you know, help improve over time how you're doing, not just for a particular page, but for a whole family of keywords and the content related to it. That was the main thing that we had to do. Once we did that, we realized that there was a whole host of benefits outside of just SEO, but we can get to that when you're ready. The challenge that we had was, how do you do this? We tried doing it manually just with one person. I described his work as like shoveling coal because it's— Yeah. Thousands of pages and thousands of keywords and trying to assign and whatever, and then go to the stakeholders and ask them, do you think, what do you think? And they would all disagree anyway. So it was very difficult to make these assignments and to govern it manually. So we decided to use AI to help us do this. So, you know, there's natural language processing, which is you basically look at a corpus of information And you extract what we call entities, in this case keywords, from those. And then you build relationships based on the probability that these keywords are together, or that in our case, that every time a product is mentioned, the keywords that are around it are extracted as well. And then we can kind of build relevance relationships between the products and the keywords. But the main challenge we had there was that natural language processing applications, like what used to be called Alchemy and is now called Watson NLU, They have a standard corpus of information that they use to help you train the system up, which is DBpedia, which is like the database version of Wikipedia. It's like all the information known to man in one database. Well, it's so generic and so broad that it's very difficult to build relevant relationships between keywords and our products. Our products might only take up a tiny fraction of that corpus. And then when you get into that fraction, it ends up being too small. So we had to build a custom corpus based on our product information, based on our knowledge center, which is like thousands and thousands of pages of information about our products. Every time a product is mentioned in that, there's keywords around it. And then also our developer worksite, which is a set of forums where our customers basically comment about the products. And then when they do that, They use language that is indicative of how they think about where our products sit in the marketplace. We mined that as well. Then based on that, we were able to build these relationships that when this product is mentioned, this keyword is mentioned. Then we just flipped it over and said, so for what keywords map to which products based on that? It's pretty simple when you ultimately unpack it. It's a lot of data and it's a lot of crunching. The basic methodology is pretty simple.

And so it sounds like the keywords you're talking about are all SEO rather than paid search keywords, and sounds like you're not doing anything with site search. Am I getting that right?

So those are additional, and we can talk about site search in more detail, but those are additional signals that we can have. So you could, you could do like paid search And throw a bunch of words against the wall, and then whatever sticks, you know, whatever actually you get, you know, clicks and conversions or whatever, that's an additional signal that adds strength to the fact that this keyword goes with this product. You could do the same thing for site search. What happens in site search is a lot of times, I think more often than not, people query the actual brand names that you have in your site. And then you look at the content and you try to associate those brand names with the content, with the words that are around it. So those are additional signals. But when we started, we used this basically our corpus for external. Like, I think Google tends to— like, the 70 or 80% of the keywords that we mine out of Google are what we call unbranded keywords. And so what we're trying to do there is associate the brands with the unbranded keywords. And you can, you can, like I said, you can use these other processes to strengthen those relationships as needed. The other thing that we did to strengthen the relationships is we ran a, you know, machine learning algorithm against it with actual, you know, supervised machine learning. So when we, when we first ran it with just the natural language processing We got about 70% accurate. We did unsupervised machine learning, we got about 80% accurate. We did supervised machine learning, we got like 90% accurate. And that's like better than anybody's ever seen. And it was a combination of focusing on the right information, you know, not being so broad, and supervised machine learning where the person who was actually supervising the learning throws out all of the, what we call, laughers, the things that don't fit. Fit, obviously to a human and not so obviously to a machine. And once you throw all those, the bad ones out, you have a pretty good clear understanding of what keywords go with what products.

Yeah, and it sounds like you actually used some semi-supervised machine learning too, but to throw what you were calling the laughers out, where— and to have that human in a loop is one of the things we found to be some of the most powerful ways of using machine learning. And I think, you know, you said 90% is really good. I think people don't understand how good that is because there's a lot of problems like this where people have the assumption that if a human being did it, it would be 100% accurate. And the truth is that usually it's not. We've had situations very similar to the ones you've looked at. There's actually been studies out there that show that if you've got enough buckets that you have to throw something into, You can actually do a task with a person and 2 days later give the same person the exact same task, and 35% of the time they don't agree with themselves. And so it's really hard to be consistent when you're a person. We're not that good at that. And if you have enough buckets, you might throw it in a bucket that's almost the same as the one you threw it in 2 days ago, but not exactly the same. So 90% is actually startlingly good. And that sounds like a— terrific work that you've done. So just getting back to site search for one more thing. One of the things that you mentioned is you use site search to kind of strengthen the things that you did. And I'm wondering what kind of allowances did you have to make because the context of site search is different? So like, if like, if somebody's searching on Google, they might search for, you know, IBM Cloud, but if they are searching in ibm.com in site search, they might just type in cloud. How did you kind of make some of the allowances so that you got the most out of site search to do that strengthening?

Well, you have to have a good system that measures, like, your site search. It measures your queries, measures and looks at the actual results that you're giving. And then you have to be able to measure, like, to what extent those results get clicked. And when people click, do they stay for a while or, you know, up to 20, 30 seconds? Or do they click something else when they actually get into those results? So you can kind of measure success in your site search. And then when they're successful, that's a good signal to you that whatever content is on that page is relevant to the keyword. If they're not successful, then you've got a problem with your content primarily. But, and then, you know, there's another action. But where those, if you look at those successes, That becomes a strengthening factor. It's like, okay, so then let's add those pages that are relevant to the— that where we had success to our corpus and rerun the natural language processing. And you can kind of— it's incremental improvement in the assignments of keywords to content, to the products that are on those pages. But it's pretty strong. Terrific.

Well, given that those success metrics are what Solo Segment uses to improve site search, I'm sure they're thrilled that you thought that they were such a good idea. One of the questions I had was that you mentioned earlier is that there are applications outside of SEO and even other kinds of search for the keyword knowledge graph. So can you kind of tell us what one of those are, or 2 of those if you have them?

Yeah, so the main one is, I think it's a content strategy problem. Like, okay, so in IBM we have kind of a 3-layer cake of marketing content. There's the advertising at the top, which is mostly The KPI for that is brand health. And then there's the— at the bottom, there's what we call the nurturing phase, where basically people have opted into our offers and all we're trying to do is target them with the next best thing that can help them progress in their journey and ultimately buy our product. But in the middle, which is the kind of big unbranded space, that's where you need to have an understanding of what is the most important content to build next? What are your target audience questions that they need answered in order to give them the confidence to work with you in the first place, to actually, you know, take one of your offers? And that's— we call that the narrative phase of the— and it's really, it's a pretty big piece of work to try to transform from a more of a transactional kind of content model to a a content model that just serves users. It's outside-in, to use our term. And it's easy enough to do that on a small scale, but when you try to do it scaled across all these business units that are all trying to run as fast as they can, it requires this governance and having the, like, all the content that is most important for that audience or that business unit to serve their audiences and then It informs the briefing process, it informs the actual content writing process and the content development and publishing and ultimately the measurement process. So it kind of— the keyword ontology is sort of the one key part, like a foundational component of our end-to-end content management system. That's a big one. The second one that we're working on is— so for social media, The hardest, the thing that marketers really struggle with is listening. And like left to their own devices, they will tend to listen around branded words or whatever. And then they'll do everything as a, I don't know, you know, social listening is basically keywords separated by Boolean operators. So, you know, whether it's an AND or an OR can make a big difference in what you get back. And it's as much an art as a science, I think. And anyway, we're building a library of these queries based on the keywords that— so for each unit, they can have what queries they need to run to get the right social listening. And it's all informed by the keyword ontology because basically it's the same keywords, it's just a different application of it.

Very cool, very cool. Getting back to the governance piece, a lot of times when people are trying to Employ that type of governance process in a large company, there's a lot of political battles, and it's your opinion and my opinion, and it can take a really long time to hash things out, and people are never really satisfied with it. Have you noticed that because you're using kind of a data-based approach, that it might take away some of those battles, that it maybe— it's more— you're almost coming to them with the voice of the customer? Rather than just your opinion or somebody else's opinion or an expert's opinion? And has that helped to make the government's process easier for you?

It has helped, although it's every time you go to— you get a new stakeholder or a new executive in a unit or whatever, you have to do the education all over again. But at least you have something to work with. Yeah, it's not just the SEO's opinion or the content strategist's opinion against the executive's opinion. It's, you know, having data and having it Be the basis of the argument resolves a lot of conflict. And, and then, you know, there are times when it's not obvious, like 2 units might say, well, it's like— so one of the things about this is that relevance is measured on a scale of 0 to 1. So like 0.75 is our threshold for whether it actually belongs in a unit or not. And like sometimes multiple units will appear, like cloud security is one of the ones that I mentioned. So then it actually becomes a job of getting people to work together and collaborate on something, and that's not always easy because, you know, like it or not, they're sort of competing against each other for the limited time and attention of the same audience. So, you know, there's always going to be a little bit of conflict at the end of the day, but at least you have a basis, a data-driven basis to make those decisions.

That makes sense. And so, and I mean, I think a lot of people, especially in larger companies that have kind of a broad product line the way IBM does, they're used to seeing that kind of intramural competition, you know, where for things like cloud security and things like that, where you wish that we were, we were trying to beat the comp— the real competition, but instead we're spending a lot of time beating each other. And I would imagine that that kind of thing might be especially helpful in governing paid search, where you really have to have, you know, one approach to manage that keyword. You can't have 2 different business units bidding against each other. Have you used this kind of approach to kind of help those business units work together to kind of really amplify paid?

Well, that's one of the next things we want to do. It's that we haven't done it yet. It's interesting that you would think paid would be the easiest because we control, you know, what goes on the page and everything, but it ends up being the hardest because—

Yeah, because it's who's we, right?

It's money, you know, and the money flows from the grassroots up, not from the top down. So like if we managed it all with the same budget, then it would be a lot easier. But ultimately, we, yeah, we have to sort of do some horse trading and, you know, you can have it, I have enough money for the first 4 hours of the day, you can have it for the— and those kinds of decisions. And that's kind of outside of the semantics. It's just a budget management issue, but it actually becomes more complicated with that. But that's something that we are working on.

Very cool. And so what are the kind of ways that you maintain the Knowledge Graph over time, or how does it evolve over time? special things that you have to do to kind of, you know, introduce changes when the data changes? Or tell me a little bit about how things morph as you're using this over a period of time.

Yeah, so the data that comes from Google changes often, at least monthly. And, you know, so words are trending and some trending up or trending down, and you have to have an additional filter for that. Like, if basically you have to continually run this thing against the keyword data. So we basically, we use BrightEdge as a keyword research platform and an SEO platform, and that's the source. We build an API from BrightEdge to this ontology. And so if a keyword is trending above some threshold, I think it's like right now 200 queries a month, threshold, then, then we say, okay, so that's an emerging keyword. Let's add it to the list. Let's, let's rerun the, the model and include it in whatever bucket it belongs in. And then let's start tracking it. Let's assign a preferred landing page to it. Let's just start tracking it for, you know, effectiveness. And let's, you know, figure out how we're going to manage it for the masthead, or what we call the internal search engine. Site search is what you call it, I guess. And those kinds of things, you just sort of work it. The thing is, once you have it built, like, that sort of day-to-day maintenance becomes a small task. It's a huge task to get it, to build it in the first place. But once it's built, it's like adding and whatever iteration.

And is it something that's comfortable in the governance process with the stakeholders, the content owners in the business units, that they just realize that it changes very frequently, or Is it something where you've actually had to work with them to get them to understand this? The thing that you, that we told you 3 weeks ago isn't as true today as it was then because this is what shifted. Has that been an adjustment for them? And is there a way that you've had to kind of work at the communication of that?

Actually, I think it changes more, like our internal branding changes faster than the keywords do. And so that becomes, that's really the bigger challenge is like we always have to be adjusting. The, like, our business units have changed and morphed and whatever. They've done horse trading with brands and whatever for the last 5 years, and it's a constant battle of, okay, so now this belongs with cloud. And anyway, so that's kind of the bigger issue. But yeah, definitely because of those, it's an opportunity to say, okay, so here, you know, I know you have this new offering or this new product, mergers and acquisitions, things like that. And here are the keywords that we think are relevant to it, most relevant to it. We ran the model and, you know, and oh, and here's some additional ones. I mean, I think that those sorts of things, we have a whole, we have a consultant for each unit that is supposed to work with them to communicate that out on a monthly basis, a regular cadence. And it tends to work pretty well. And, you know, there's always going to be mess. That's just kind of big corporations.

Those are good words to end on.

We're out of time for today.

But James, I really want to thank you so much for this. These are great ideas. And especially I want to thank our audience for joining us. Later this week, we'll send you all a link to the recording of this webinar, and we also invite you to mark your calendars for our next Biznology interview, Top Trends in B2B Marketing with Ruth Stevens, on July 10th at 11 AM Eastern time. So sign up for it now at biznology.com while you're thinking about it, and we hope to see everybody back here then. Bye everybody.

Bye, thank you.