Gaga for Google Custom Search Engines Part 1 - REPLAY

Happy Friday, everybody, and welcome to a reboot of Google Custom Search Engines. If you were like me and almost bailed, or actually bailed last Friday when we tried to broadcast the first episode of Google Custom Search Engines, I would completely understand because the video froze after about 6 minutes. The audio, fortunately, went really well, and also we had the inimitable Glenn Guttmacher taking notes— nay, not taking notes, actually summarizing very succinctly everything that I had to say. So the audio and the notes are pristine, and we're going to use those same notes today. However, we're going to re-record the video because there's definitely something to be said about being able to see how I do some of these things. I want you to know that this is really simple, okay? It looks complicated on the surface. No, I'm not being facetious. It's something that can be intimidating, but custom search engines on Google are really easy to set up once you understand what the basic components are. So today I'm going to walk you through the 3 basic components as well as how to get it set up. Let's go ahead and jump right into it. First of all, this is free, okay? You go to google.com/cse, as in custom search engines, that's google.com/cse, and from there you'll be able to see the blue icon. It should be a blue icon that says, let's go ahead and, um, get you started on a custom search engine. If you have that, then you can just click on that button. If not, it might take you to a window that says, here's a list of your custom search engines. Maybe you've experimented with it before or whatever. Bottom line is you actually need to have a Google account. Doesn't have to be Gmail, it could be company Google Apps account if you have that, but as long as you have a Google account, you can log in and create a custom search engine. You can just create a Gmail account specifically for this purpose if you want. You can also use your regular email address and create a Google account, but once you do that, you can go to google.com/cse, and that's where you'll be able to create your very first custom search engine. So let's go ahead and take you there. Here we go, google.com/cse. So this is what it's going to look like if you've never created one before or logged in before. and up on the right-hand side you'll see a little Get Started button. Now if you have one and have created them before, I'll show you what that looks like in a minute. Let's go ahead and get started. It'll ask you to log in with, with your Gmail account, your Google account, and once you do, it'll take you to a list which is where you have all of your previous custom search engines. Yes, I know, I, I I do have a little bit of an addiction problem here, and believe it or not, it's, it's worse than it seems. I have like 3 other Gmail accounts. This is my quote-unquote company or business account. I have my Shelly Gmail as well and 2 others, and they're all full of custom search engines. There's no limit to how many custom search engines— actually, there is a limit. You can have as many as 5,000 entities with a Google Custom Search engine. What is an entity? An entity is a website that you're searching, a custom search engine itself, or refinements or synonyms. So think of it as an element. You can have up to 5,000 elements. So that means in one custom search engine, you can have 5,000 sites, or you can have 5,000 search engines with one site each, basically per Google account. Of course, you can create multiple Google accounts, so that's why I said virtually no limit. So let's go ahead and begin. If you don't have any, then you'll see a little blue button that says Get Started. That would be, you know, up here somewhere. I'm not sure what that looks like, and I don't want to waste your time finding it. You'll see it, you'll click on it, and you'll end up basically here, which is how you start. The first thing that you need, absolute first thing that you need, is a website, any website, whatever you want. You could put LinkedIn.com, um, or you can put the website of an association, a conference. What you're doing is you're going to use the Google search engine power to look through the content of one or multiple websites. You can also search the entire web with some additions and refinements, which is a different way to use custom search engines, but we'll get into that. After I introduce the basics. For now, let's just start with one website, any website. Of course, you can add 3, 4, 5, whatever you want in here, but really just need one. Once you've created them, you'll see a list of all of the ones that you have here on this dropdown and also here. Now, in the last search we did, we started by creating a demo LinkedIn search and just added one one single website which was LinkedIn, and then I showed you that you might want to separate LinkedIn into pub and in so that you show both of the results that are still with the old public folders as well as when someone creates a quote-unquote vanity URL, and then we added a couple of other things just as experimentation. So basically, put in your first way, LinkedIn.com, then you can give it a name, This name doesn't really mean a whole lot. It just shows up on the list. You can give it a description. Nobody but you or admins see this description, and then you can add keywords. The only significance of adding keywords here is if you put keywords here, those words will increase the ranking of pages that come back as a result, but it won't change the search itself. So the search will still be the same search. However, pages with those words in them will will be increased in rank as far as the way you see the results. So you can leave this blank if it's going to be a pretty general search engine, but if you were doing a search, for example, for chief legal officers, that's my other search engine that I've been playing around with lately, then you put in keywords here like general counsel and chief counsel and all the other words that relate to that. That means that those profiles will simply be given more weight in your search results. Then you see the public URL. This is what you can share with people. This is how you can search— that is embedded HTML, so you can embed the box on your CRM that allows you to do a search, or you can just simply go to the link. Like I said, there's not much here, it's pretty unassuming. The next thing that you have to decide as you're building is do you want to have image search as part of your search or not? The useful component of image search is of course faces or groups of people, so if you want that, great, if not, turn it off and it's just another tab that shows up if you do want it. Safe Search is where you decide whether or not you want any content that may be considered adult content. If you turn that off, it's simply going to bring up back anything. If you turn on Safe Search, it will eliminate pages that have, let's say, ratings of above R, right, adult content. So it might actually limit your results a little bit, which is good. The region doesn't matter much unless you're specifically narrowing it down to a particular language. If you choose United States, it is going it's going to try to limit your results to only websites that are potentially in the United States, but it's not going to eliminate things that it's not sure about. So if it doesn't know, it's going to leave it in. If you restrict the search results to a region, then it will only bring back content that is in the United States. You may not want to do this if you have a search that might include, you know, anything outside of the United States. Think Canada or anything else really. Language does restrict your results. Of course, if your native language is Spanish or something else, you would want to change that. Otherwise, just leave it at the default, which is English. And then finally, we have additional websites that you can add. What I did here is I added a list of the top 100 nonprofit organizations because I've been working on a nonprofit CLO search. And so here you see a whole bunch of websites about 156 that I added. Now before you think, oh my, Shelley, that looks really tedious, well, it's not as bad as you would think. Actually, there's a simple way if you upload websites in bulk to add a whole bunch of them if you have them, say, in a spreadsheet. Or you can do what I did, which is to go to a website. I'll show you here just a second if I can do that. And It is App Tim 100. So here's just an example. This is a website that I— that was recommended that is a list— that's not loading for some reason. Okay. Why isn't it loading? Sorry about that. No idea why it's not loading. Okay. While I wait for this page to load. Streaming, just don't know why it's not coming up. The other thing that I can share with you is that you can also have these in a spreadsheet, for example, and that's something that I have. I've kind of over the years collected a bunch of different websites and domains and things like that, so that's another thing that I have here. Let me, let me share my screen. Share with you once again so I can show you what that looks like. So you can see here, this is an Excel spreadsheet, and it's pretty basic, just has the URL and then the type and some labels. So for example, I have associations, cloud documents, so I can sort them or I can filter them. Let's do cloud documents, for example, and then I can just select this list and paste it directly into the custom search engine. So you don't have to worry about doing doing it one at a time, meaning you don't have to add each individual search one at a time by clicking on add, you can add in bulk by clicking include sites in bulk, and then just paste the list here. Now, on the last time that I did this, when I pasted it from Excel, it put quotation marks to the beginning and the end, so I would probably use Notepad, that way it doesn't include some weird HTML, and I didn't know that it was doing that. It didn't do that in the past, but it started doing that recently. Probably a new version of Excel, so just copy and paste it from a notepad. Each website, each URL is a separate line, or you can of course include them individually. Now you also notice that I have some patterns. I'll explain that later on in the advanced series, but patterns allow you to match various websites using essentially, you know, before and after the, the main domain criteria. So let me just scroll through here and I'll show you. A lot of these are simply websites from that particular list. The reason you see a red X is because if you're going to use JSON API, they will only allow you to have 10 websites. But since I'm not doing it, it doesn't matter. I'm using the, you know, the public link. So they're X'd out, meaning that they won't come up in the API because they're exceeding the limit of 10, but they are being included in this search here. Let me see if I can get to some of the others, and here we go. Here's a few:.org,.ngo, and directories. So these are under label. What label is, is essentially the refinement tab. So once you have a few websites, the next thing that you want to do is create refinements and synonyms, and that's it. We've covered the basics. The 3 basic components are websites that you search, that goes in here, then refinements and synonyms. To get to those two, you're going to go to search features. From search features, you see a few tabs across the top. The second and fourth one are the ones that are of interest to us. The second one called refinements. Refinements are also labels because refinements work in two different ways. The first way is by aggregating a collection of websites. So let's say that you have in my case, 100 websites that are all related to the same topic, or in this case, there are 100 websites of nonprofit organizations. I can use this label and search only those websites so that when I go out and conduct a search, as I'll show you here in the little test window, you'll see that there's a Raptim 100 list which is going to search for my keyword in only those 100 websites that are quote-unquote labeled by that refinement. The second way that you can use a refinement is to add additional keywords at the end of the refinement, and that instead of restricting the search to only the sites in this label, you can give priority to the sites in this label and just simply add keywords here much like you would on a search. So if you were looking for chief legal officer, you could put in here chief legal officer or CLO or let's say general counsel. This is now a different kind of refinement that actually adds keywords to your search. Let's call this, um, we'll just call it CLO short, and that's what the label is going to be called when we update the spreadsheet or that the search engine. I'm sorry. So now I'm going to go to my programmable search engine here, and I'm going to type in, let's say, World Wildlife Fund, hit enter, and let me zoom so that your eyes don't get crossed here. I can then— this is the all results, which we haven't talked about yet, but I can then click on the CLO refinement that I just created, which will append those keywords to my search. You can see general counsel, chief legal officer, not seeing any CLOs yet, but there you have it. So that's a very good way to use refinements. The other one is going to search for World Wildlife Fund in the Raptim 100 list. So only the list of websites. Now notice down here that you see two things labeled org and Raptim 100. These are what's called faceted search. So when you choose a refinement like the NGO, you can also activate what's called faceted refinement. If you turn that on, then the results will show whether or not any of the results on that page match that criteria. In this case, I have the org turned on to facet refinement, and what this does is it restricts my search to only websites that end in.org. Any website ending in.org will be marked here as that label. Of course, most of the nonprofits are.orgs, so a lot of them are coming up. So that's the refinements. Once again, to recap, you can do refinements that are a collection of websites, so only these websites, or you can do refinements that add keywords so that you have essentially a hidden search string that you don't have to worry about copying and pasting or remembering. Both of those can go in there. Folks are asking if you can access this later. Yes, the recordings are definitely live. I'm afraid that it seems to be freezing up again. Is that the case? Are we freezing all over and the video is locking up? I hope not. Hmm, okay. Well, if you want to come back later and check out the recording, just go to my website and you'll see links to the video on LinkedIn, on Facebook, and on YouTube. In the meantime, let's go ahead and return to our regularly scheduled programming here. So we started out with talking about what this is for. It's for collections of websites. You can do that by adding some websites that you want to search into the site list. Then you can create refinements that clump websites together, and then you can create synonyms. Those are the three components. So let's talk about synonyms next. Synonyms are essentially your own custom dictionary, much like you would create in an OR statement. So here for CLO, I chose words like, or phrases like corporate secretary, chief legal, legal officer, general counsel, etc. So whenever I search for CLO in my search engine, those keywords will be added as well. It will expand my search to include those keywords. I also did the same thing with attorney and I did the same thing with NGO to expand to non-governmental organization. Some examples of synonyms you can create are names of cities, area codes, collections of zip codes, collections of job titles, collections of companies. And by the way, you can actually give your synonym a nonsense name so that it only works to expand those criteria and doesn't search for that, let's say, extremely common word. I'll give you an example. Let's add a synonym called NGOs1, NGOs1, and for this we'll search for just the keywords World Wildlife Fund, then perhaps Planned Parenthood, and I don't know, Greenpeace. And let's try World Monetary Fund. Yes, I know it's not an NGO necessarily, but just for show, just try these for now. I'm going to do NGOs or NGOs1 actually. That gets added to my list of synonyms, and now when I go and search for NGOs1, it's going to append those keywords to my search regardless of where I go and execute the search. You can see on the Raptim 100, it won't find them because those words aren't found on those websites. Try it with the other one, not getting a whole lot of results, so probably I was a little bit too aggressive with refining there, but you can see how it works. It simply just expands that keyword to include these others. Again, job titles, city names, for example, city names and area codes and so on. So think of synonyms as your custom dictionary, your custom ontology. Those are the 3 main components. Once you have those 3, you can then start playing around with different concepts. Let's go back to the setup and I'll show you what I mean by that. We could create collections of websites and add keywords to those. So say for example, when I'm searching the Raptor 100, I'm also going to add a particular keyword so that we'll search only those websites and add that search string. There is a limit to how many keywords you can put in that search, so you don't want to go crazy with too many, but it does help refine results if you're getting too many. Another concept you can play with is to expand your search to include results from the entire web. As you can see, I've already done this with this search engine, and I want to show you the difference between the results. So let's go back to our original search here, and I'll do a search for attorney. And actually, I'll just use attorney and then maybe go to— that's a lot of results. Let me try one other thing. We'll do, uh, nonprofit attorney and nonprofit. Okay, still a lot, but not bad. So in the all results here, I'm going to zoom in so you can see what I'm pointing to, are these 730,000 results. Now don't take this as gospel. It certainly is not 730,000 results. A Google Custom Search engine is only going to produce a limited amount of results. As you can see, it only goes to page 5. So you won't see all of that. For that reason, it's very important that you narrow your search down so that it only gives you maybe 200 to 300 results. But for the sake of the demonstration, I want to show you what happens when I turn that off. Right now it's 730,000 results when I search for attorney and nonprofit. If I go back here and turn off search the entire web, now my search engine will only search these 156 websites and nothing more. So when I come here and search for all results, I probably should reload it. Huh. That's not a big difference. I'm surprised. It only went down to 700 and— oh, look at that. Okay, let me try and reload this from scratch because I'm getting 3 different results each time I try. All right, one more time. 706,000. So it took out almost 100,000 results, which may not seem significant, but it is because now I'm only searching the.org websites, nothing else. Because that's actually one of the refinements in my, in my search engine. Wrong way, here we go. Probably closer to the end there. I think it was one of the last ones that I, I mean one of the first ones that I added. There it is, right there. Because of that, it's including a lot of websites. If I remove that, it would be a very, very narrow search. So now that I've showed you that, I can also show you that you can remove websites. That's another concept that you can play around with. So if you say search the entire web, and now all of a sudden you're getting way too many results, and you're like, wait, this isn't working, Shelley. How come I'm getting about the same kind of stuff that I would get from a regular Google search? I don't need custom search engines. Well, that's because you're probably including too much, And so one of the things you can do is use refinements to narrow your search. Another one is to use synonyms, and the third one is to eliminate websites. You do that by going to the advanced, which is right below the websites here. And one of the things I always do with my custom search engines is remove all of the job-related websites. Remember when I said I was going to talk a little bit more about patterns? We're going to do a deep dive on patterns and how they work. On our next session, but for now, suffice to say that anything that has the word vacancies or career, job, jobs, examples, and samples in the URL is being removed as long as it's on a.com website. So basically, almost all the job websites are being removed. If there's a website in particular that is offending you a lot, you know, it's coming back over and over again, maybe like Glassdoor or Indeed, and it's nothing but job postings, you can also remove that. In this case here, I removed LinkedIn because I have another custom search engine that searches just LinkedIn, so there's really no point in having LinkedIn results again over here. I'm separating my web search results from LinkedIn. I've also removed a few of the most common job board websites, as you can see, as well as those patterns. You can remove your own company, you can remove a particular hands-off organization, etc. So let's go back to our search, I mean our presentation, sorry, and talk a little bit about what we've done. Google.com/cse, build your own search engine that searches only the websites that you want or the entire web, but using refinements and criteria that you specify, including your own dictionary of synonyms. You then add websites, you could add websites such as social networks or cloud document websites or online resumes. You can add competitors, you can add associations and conferences, standards bodies, any collection of websites one at a time or in bulk. Then you create synonyms, for example, similar job titles, similar companies, similar skill sets, tools, the names of degrees that are comparable. I used to do a lot of computer science searches at Microsoft and at Google, and I can tell you at least in my recollection there were 13 different ways to say computer science degree. So I had a collection of keywords that meant computer science, like, you know, BSCS, BSEE, BSC, B little sc, and so on. You can also use abbreviations such as corporate nicknames or model numbers, etc. and then you can add to your refinements by adding additional keywords in the refinement, or you can even go into the advanced settings of your custom search engine and add words that have to be included in all results no matter what. This can be pretty limiting, so use it wisely, but what it does is it eliminates too many results when you're opening your search engine to the entire web, and it reduces when you have too many results is what I meant to say. So you now have a much better set of results and don't waste a lot of time on dead ends. Here's a couple of refinements that you might want to try. You could use a refinement to search only email addresses. Hint: if you have a Gmail, Hotmail, or Yahoo account, you're part of about 90% of the population. So let me show you how that would work. You could easily create a refinement that just searches websites that have webmail accounts. But for that, you would just add here that they must have either gmail.com or yahoo.com or hotmail.com. Why not go crazy and let's also add the me.com for Apple, and why not, let's also add live for Microsoft or Outlook. That's probably better, outlook.com. So now I'm going to search anything that, that I search on the entire web using this refinement. I'm going to search for these email addresses, hit OK. Let's go and test it over here in our little window and say we want We'll do chief legal attorney NGO, and I'm going to click on webmail, and you'll see that it's only going to bring back pages— I know that's pretty small here— that have Gmail accounts in them. Sorry about the size of that. Let me go to the public search here and do that instead. They're a little bit better. I'll just use this one here. No, didn't bring any back. Interesting. I wonder if it's because of the nonprofit or something. Trial and error doesn't hurt. Ah, okay, so this is telling me that something's wrong with the refinement. Whatever it is, I'll have to troubleshoot that later. Let me see if I made a mistake here. Hm, I don't think I did, but you never know. So how about I just do this? Get rid of the asterisk. I meant to get rid of the signs as well. If maybe that's messing it up somehow. Let's try that again. I guess so. Nope, nope, not at all. Interesting. Well, let's figure this out later. Actually, I think what I'm going to do is just start a new search here. Click and attorney. It just hadn't had time to load or something. There we go. Okay, now it's giving back results. With Gmail. I should have just reloaded it from scratch, but there you see Gmail. That's gonna be the most common one, by the way, probably followed by Yahoo, probably followed by Hotmail and Live.com. If you wanted to get much more than 90%, it would be really, really difficult. You'd have to add a lot of the ISPs and so on. Certainly you can add a few of the, the Apple domains, like I think it's— there's me and there's iCloud. I think there might be another one. So there you have it. Those are the basics of a Google Custom Search engine. Just to give you some food for thought, you might try to add refinements for file types or for particular types of content, like pages that have RSS feeds, that would be a blog, or you can use natural language phrases. I use that sometimes for competitive intelligence. When one person reports to the other usually, and there's a comment or a paragraph about that online, you might see the phrase "reports to" or "reporting to" or "reporting into." So that's a natural language phrase that I might use as a refinement in a certain type of search. So here's just some ideas. This will be in the notes. You can try. I'll show you that file type one because I don't think I have that one added here. So let's add that real quick. That would be a refinement that you would add here. And let's do PDF, for example. And we'll do— oops, wrong place— file type PDF, which should search only PDF websites. PDF websites, I'm sorry, websites that have PDFs. That's what I meant. Ah, again, I'm getting that no results. I'm realizing that I have to click on this public URL basically every time now in order to get the search to reset. Okay, never mind that then. All right, wait, did I do search only these websites? Is that what happened? Sometimes there's a learning opportunity for me that I think I— ah, Yep, okay, there's no websites. So by logic here, you're saying search only the sites with this label and nothing has that label. And so this is a mistake that I made. You have to say, give priority to the sites with this label, otherwise it's gonna try to search only those websites which you have none labeled and hence the problem. So I bet now it's gonna work. Little details like that sometimes trip you up. By the way, that might be one of the reasons people get frustrated creating these custom search engines. So maybe keep a little checklist that says, okay, I got to make sure I do this, this, and this. But I don't want to overcomplicate things. The basics are add a list of websites, create a collection of websites with refinements, add some additional keywords with refinements that are also essentially labels, and then some collections of keywords that are synonyms. Don't get confused between refinements and synonyms. A refinement that has keywords in it is searching for essentially a search string. Keywords is expanding— synonyms, I'm sorry, is expanding keywords in every search that you use on any of the tabs on your custom search engine, even if it's not one of the refined tabs. So a synonym is universal to your custom search engine. That's the main difference. So with that, I'm going to turn to the chat here and see if we have some Q&A that might be answered. Do the synonyms need to be in quotations? Good question. No, they do not. Synonyms do not need to be in quotations. In the synonyms, just separate the words by commas. So General Counsel, comma, and then CLO does not require quotation marks. So that's a good one. Okay, looks like we have some comments that the video is not frozen. I'm very relieved to hear that. Looks like that was the only question that came in. That's a good one. All right, folks, so with that, I bid you happy weekend. Come back next Friday when we're going to dive into a lot of the refinements and go into some very specific examples on refinements and patterns, as well as how to use the remove websites. So the concept of the, the second session on Google Custom Search Engines will be manipulating URLs, basically refinements, removals, and patterns. And then the third one will get into some interesting ways to use synonyms. And then the fourth one will be the advanced settings. There's a lot more you can do with advanced settings, but I don't want to get into that too much because I don't want to lose your attention, and I would like for you to have some practice using and creating some custom search engines before we go into actually creating advanced settings. And then the 5th one in the series is going to be how to use both of the extensions of custom search engines. One of them is the schema.org, which are HTML schemas, and the other one is the entity search. So that's, uh, let's just call it the most advanced you can do with a custom search engine. And I'll also throw in a hint, hint, hint, other custom search engine that you can use that is not called Google. We've probably covered a little bit of that in the past, but it's worth mentioning. So that's this, the next, uh, 5 episodes in the series. Hope you have a great weekend. Thank you very much. Bye-bye.