Mike Moran explains how search engines work: they crawl pages, build an index of words and URLs, then rank results using on-page factors like keyword density and off-page signals like inbound links. The same query analysis powers both organic results and paid ads, which draw from separate databases but follow the same fundamental process.
Chapters
Hi everybody, and welcome to Lecture 2 in our series. This one's called How Search Works. We're going to look at how a few different things work: how search engines work, how search marketing works, and how searchers work. We'll do all of those things in this lesson. I'm Mike Moran. I'm the co-author of your textbook, Search Engine Marketing Incorporated. I'm a former IBM distinguished engineer who now serves as chief strategist at Converzion, which is a digital media marketing agency based in New York City with clients worldwide. And we do many things in social media and in search marketing for our clients. And you can learn more about me at my website, which is at mikemoran.com. Let's start with chapter 2 in our book, How Search Engines Work. And although this material is taken from chapter 2, I just want to warn you that there's a lot of stuff in chapter 2 that we're not going to talk about in the short time we have for this lecture. So it makes sense for you to look at your book as well. Let's start with understanding how organic search works versus paid search. So when you type in a keyword such as digital cameras, as was seen here, you see 2 different kinds of search results. You see the kind that's in the orange box, which are the paid search results. You also see the other results that are in that middle column to the lower part of the page, which are known as the organic search results. So the first thing I want to do is just talk through what some of the terminology is. So I used some terms already. Keyword is what we call the words that are typed into the search box. In this case, digital cameras. So some people will distinguish between whether there's one word or multiple words. They might call one-word search terms keywords, but they might call multiple-word terms key phrases. I don't worry about such distinctions. Whether it's one word, 2 words, or 20 words, I'm gonna call them all keywords. And so, but if you hear that other terminology, you'll know what they mean. And keywords are critically important because only when you know what the keyword is that someone is searching for does it make sense for you to say, I would like to be the number one result for that keyword. You don't necessarily need to be number one for lots of keywords because You want to be number one for the keywords that people are going to use who are likely to buy your products. And so focusing on what keywords people use is a critical part of success of search marketing, regardless of whether it's organic search or paid search. So let's start with organic search. So those items that you see in the lower part of the page, so They're in the lower middle part. Those are the organic search results. And you'll use other— you'll hear other words used for organic search results. You sometimes might hear them called natural search results. And you might hear them talk about what you do to get your pages there: search optimization or search engine optimization. All those things mean the same thing. It means trying to get your pages to show up in that part of the search results screen. And we're showing a screenshot of Google, but we could have been showing any search engine. There's no reason why the search engines had to make their pages look the same, but over time they've tended to do that. So typically those organic search results show up right where they show up with Google. And the paid search results are above the organic search results and down the right side of the page, as you see on the screen. Now, time was that different search engines had different looks to their pages, and that could happen again. At one point, paid search results were sometimes intermingled around organic search results and not even identified. But you can see in this screenshot that you can see the word sponsored links in a couple of places that identify which are the things that are paid and which are the things that are not. And so that's pretty standard now, and I think search engines would come under a lot of fire if they ever tried to go back to not identifying the paid links and also to intermingle them to try and possibly confuse searchers as to which is the organic and which is the paid. So you won't— I don't think you'll see that again, but that's something that was true in the past. Now, organic search is kind of the librarian's answer to the question. Paid search is the advertiser's answer. So paid search is all of the things that someone has paid to show you. Although they're not paid in the traditional way that advertising is paid. If you think about how a newspaper or a magazine works, you pay to put your ad in the issue, to have your ad show up on the page. Same is true for TV and radio. You're paying for a certain amount of time on the air regardless of whether anyone buys anything from you or not. And paid search is a little bit different. You're still paying regardless of whether people buy from you, but you actually don't pay anything to have the ad shown. What you're actually paying for is to have the ad clicked. So you're paying a certain amount of money— it could be a nickel, a dime, a dollar, could be even much more than that— every time someone clicks on your ad to go to your website. But in return for that, you get to control what's on the screen. So those titles that say 20% off Nikon camera or digital cameras on sale in the lower right, Those are titles that you chose, and the words under it that say things like find great deals on digital cameras, those are things that you chose as well. Organic search is a little bit different. Organic search is kind of the librarian's answer. So it's what Google or another search engine thinks is the absolute best answer to that question that the searcher is posing. And so when you look at a title in that in that organic search result, such as digital cameras unbiased digital camera reviews prices and advice, what you're seeing there is the title of that webpage. So the company whose site created that page gets to pick that title. But what's under it, where it says for the digital camera buyer, comparison of digital cameras, etc., that's something that Google selected based on something that occurred on the page. So the, the company that owns that website may have put lots of words on that page and Google picked out some words that it thought were the right ones. Usually they're words that contain the keywords, sometimes the first occurrence of the keywords, or the place where you can see the keywords multiple times. So you don't have quite as much control over what shows up as you do with paid search, But the good news is that you don't have to pay for anything. You don't have to pay for the clicks. You don't have to pay for anything to the search engine. As we'll find out, with any kind of marketing, you end up paying something, but it's not to the search engine. Let's start by looking at how organic search works. So it starts with a query. Now query is yet another name for those keywords, those key phrases, those words that someone types into the search box. So when a searcher enters those words, the search engine analyzes the query. So what is it looking for? So in Western languages, it needs to figure out what language it's in, how many words it is, are the words spelled right. If they're not spelled right, they're gonna be— the searcher will be presented with a did you mean, where the search engine has looked and seen that there are many more popular words that are spelled almost the same as what was typed in. Eastern languages, you still have to figure out how many words there are, but because there aren't blanks used to separate words in Eastern languages, that's a little bit more difficult effort. And for that reason, it's pretty important to figure out what language these words are in. And all those things are done for the search engine, or by the search engine, as people type in those search words. Beyond that, the search engine also looks at different forms of words. So usually the search engine will treat upper and lowercase versions of the word as the same. Search engines also look at different verb forms, so the word run versus runs are usually treated the same. And sometimes the words don't even look like each other, like the words be and is and were. They're all the same verb form, but they don't look very similar. Singular and plural is another thing that Western languages also have to deal with. Where search engines will typically use house and houses to be pretty much the same thing, although there are some situations where it might distinguish. But again, those words don't always look the same. Mouse doesn't look a whole lot like mice, but we know it's the same word. Asiatic languages don't have concepts like upper and lowercase or verb forms or singular and plural, but they do have some other issues, and search engines use this phase of the search to look through all of those kinds of things. They also look at things like stop words. So there's certain words that don't have a lot of meaning, like in English the word the. They look at phrases that occur together. They also look at something they call antiphrases, where if you— if someone types in, what is the address of the White House, the words what is the address are maybe not all that useful. Certainly the words what is aren't useful. Address might be useful. So those words are antiphrases, and those will be removed for the purpose of figuring out what the right search words should be. The next step is to look at the search index. So once that query has been analyzed, then it looks in a file called the search index. It's a database You can think of it as the same kind of thing as a spreadsheet or a fielded database, where the first column is every word that occurs out on the web, and the second column would be a list of all of the web addresses or URLs where those words appear. And so we'll talk a little bit about how this file is created. But this is how the search engine is able to provide answers so quickly. It's actually preprocessed all of the data on the web into this file. So if you type in digital cameras, it looks down its first column of the spreadsheet to find the word digital. And then it checks to see all of the pages that have that word on it. Then it looks at the first column again for the word camera, and then it finds all those pages. Then what it does is looks to see which pages have both on them. And so that's a very simple explanation for how a search engine works. There's a lot more to it, as you might expect, but that's how it uses that search index. Now, one thing that you should understand from just this simple explanation is that if your page is not in a particular search engine search index, it will never be found. And so even though your page might appear on your website, you can go to it with— by typing in the URL, you can look at it in the browser, it's a live page, it's out there. If the search engine hasn't placed it in its index, it will never be found whether the words that people type in appear on that page or not. So that's a critical thing in terms of getting organic search success, is the first thing that has to happen is to make sure that your pages are in each search engine's search index. The next step is to look at how the rank— how the search results are ranked So if you think about it, getting back thousands or even hundreds of thousands of pages in organic search is not all that helpful unless the best ones are listed at the top. There are very few searchers that even go to page 2 of search results, so getting the best results on the first page is a critically important function of the search engine, and ranking is one of the most intricate features in search retrieval. So, and each of the search engines, it's kind of their secret sauce. They are quite concerned about making sure that they have the best ranking algorithm they can, and they don't share with anybody exactly what goes into that ranking algorithm. But we do know some things. We can't tell you exactly how they do it, but we can tell you a lot of things that we do know just from experimentation and from reading patents and from things that have been publicly stated. So there's a couple of different parts of a ranking algorithm that uses different factors to decide which things, which pages should come up higher than others. So one of it is what's on the page itself. So if you look at the text that occurs on a page, there's a lot of different factors that you can think about that concern the text. Now the way I'll describe these, they sound like they're all separate, but they're kind of not. So there's something called keyword density. So keyword density is, is how frequently they occur on the page compared to how many words are on the page. So if you have a very long page, you'd expect to see the keywords occur more frequently than if you had a very short page. There's also keyword prominence, which says where do those words appear. Do they occur in the title of the page? Do they appear in the first paragraph of the page? That would lead the search engine to conclude that the page might be more about that subject than if they only occurred very deep in the document. And so, and both of these things together, and with a lot of other factors of how you analyze text, give the search engine kind of a theme of the page. So not only the search words that you're typing in, but also other words that the search engine knows to be related to those words, that tend to come up in common parlance when those words are being used. All those helps the search engine to understand that the content is about that kind of information. And so it uses all sorts of techniques that can be quite intricate. One of the most popular ones is called latent semantic indexing. And so what that uses is the ability to understand that when words tend to appear next to other words in normal usage. And if you start to see words occurring together that's unusual for the language, then you know that that's probably a place that has a particular concept. And so when people type in those words, or they type in words related to those words, that can help the search engine know those are the right pages. But that's probably not the most important part of a ranking algorithm. When the search engine ranks the pages— and sorting is another way of thinking of what that means. So when it sorts those pages, when it ranks them, it can't always tell which pages are higher quality than others. And so in order to do that, it uses links. There are many things that search engines can use to decide quality. It could look at how many times pages are viewed. And now that social media has become more prominent, it can look at how many views you have of a video, or how many people comment on a blog post, or how many people retweet something. And so all those kinds of things can be used as factors and many people think they're beginning to be used as factors. But the thing that search engines have started with was the link. And so looking at how many other sites link to that page and how good those sites are by seeing how many sites link to them, that's what gives the search engine the most positive understanding of which pages are high quality and which ones are not. The next step is to decide how to display the search results. So you figured out which page— which pages you want, and you've also figured out what order they should appear in. Now the search engine needs to decide how to show them on the screen. And so there's 2 pieces of information that show up for organic search results. The 2 most important pieces are the titles of the pages and the snippets. Now titles, you might think from looking at the webpage that whatever is at the top of the page in bold print is the title. Because that's what the average web user would think. But that's not the case. For those of you who have an understanding of the markup language that creates webpages, which is called HTML, Hypertext Markup Language. It's actually a particular tag called the title tag, which has in it the data that shows up on the screen. So when you look at a search results screen and you see usually it's blue text that's underlined, that's the title of that page. That comes from that title tag. And what's under it is called the snippet. It's something the search engine extracts from the page or sometimes from the description tag. For that page. And so it's usually something that's in the body content, although sometimes it can be the description tag. And what it's looking for is an occurrence or multiple occurrences of the keywords that the searcher typed in. So it's extracting a portion of the page and it's putting that underneath to give the searcher kind of a preview of what would— what should be on that page. And both of those things are contained in the search index. So it isn't going back out to the page to look at it. In the search index is the information that it needs to pull out the title and the snippet. And so it goes and for the top 10 that appear on that first page, it pulls out from the index titles and snippets and then gets ready to display that page. The last step is to actually construct the screen. And so It formats all the information to go where it wants it to go on the screen. It applies the right stylesheet, just like any webpage, and it presents that on the screen using the titles as hypertext links so that people can click on them to go to the actual underlying webpage that came from the original site that went into that search index so long ago. So, it looks like magic. It happens really quickly, and the real magic is how much can be done in a split second. There are many other techniques that might allow search engines to do a better job of finding the right pages. The problem is that with today's computers and today's understanding of how search algorithms can work, so far we can't use some of those other techniques and still return the search results as quickly as people expect them. I mean, it seems almost strange that answers that people used to be willing to troop down to the library, spend an hour of their time looking up in order to get the answer to the question, now they wouldn't wait 10 or 15 seconds to get that answer from a computer. But the truth is they won't. Unless you can do it in under a second, people get impatient. They, they back up from the page. They'll go somewhere else. They're very impatient with web pages. They need something to happen right away. So the real magic of search engines is not that they find the things, but that they're able to do all this kind of processing this quickly. So let's go back to the question that we left hanging earlier, which is how do these pages get into that search index in the first place? Well, it uses a program called a spider. This is actually geeky humor because the spider crawls the website. So, what that spider program does is it goes, for example, to the homepage of a website and it looks at all the words on that page. So it remembers this web address and it remembers every word that it found on that page. And it constructs that spreadsheet-like table that we talked about that has all those words on it. And it goes and updates each word's entry in that index. And it identifies that this new page has those words on the page. Then what it does, it looks at the links that are on that page that go to other pages. And it goes to each of those pages in turn. And it scrapes all of those words off those pages as well and updates its search index. But as it does it, it remembers every link that it's seen. So that's how it can both find the words on the page and also know which pages have the most links to them and where they're from and how good those links are from other sites. And so it does each of these things in turn and it does this constantly over and over again every minute of every day. The search spider for each search engine is out updating its search index. Now for some pages that change all the time, the spider comes back very frequently. But if it comes back to your site a week later or even a month later and it finds that the pages haven't changed, it'll then start to come back a lot less frequently. So the more often the pages change on a site, the more frequently the spider will visit. But if you wait long enough and you've got at least one other site that's linking to yours, the spider will eventually discover your site and it will index your pages. So we talked about how the spiders find the pages, but we didn't talk about is what they see. So what do the spiders find when they go to those pages? They don't really see everything that we human beings do. So if we go to a page, we can see that there's different colors on the page, different fonts. We can see the images and other pictures. But what we, what we forget is that the spider is just a program. So it doesn't see all those things. What it sees is the HTML code. Markup that goes on the page. So it sees all the different links, it sees the title tags, it sees the other types of tags that are on there, but it doesn't really know, for example on this page, that that is a picture of a book cover. It just knows that it's an image file, and it may know that there's a certain text that's associated with it that you might see when you mouse over it. But that's all that the search engine really sees. And so we need to remember that even though we have all sorts of information coming at us when we look at a web page, search engine mostly has just the words. And so it's critically important that we keep that in mind when we're doing our search optimization. Now let's shift gears and look at how paid search works. And a lot of this is going to look very similar. So with paid search, we still have to do a lot of the same things that we did with organic search. So the step of analyzing that query, figuring out how many words it is, figuring out what language it's in, all those kinds of things are the same. And in fact, the search engine does it once each time a search query is entered. And so it doesn't do it once for organic and once for paid, it just does it once. So those steps are the same. But what happens next is different. Instead of looking in the search index, paid search queries look in the ad database. So this is the place that retains all of the advertisements, all the paid search advertisements that have been placed with that search engine. So if you want to advertise, you need to put your ad in that database. And just as with the organic search engines,
