Shally's Edge: Search Engine Bias (6 min)

So folks, I want to talk about something that affects all of us that use the internet, and that is that search engines and search algorithms have their own agenda. They're really not neutral. In fact, they— the way that they measure results and rank results is purely shaped by profit motives. In other words, they're trying to sell you advertising. As a matter of fact, you are the product. What they actually are selling is your eyeballs for their advertisers. So are, are we really able to trust what search engines say? How much of what we find is actually dictated by what a search engine wants us to see versus what actually exists? Let's talk about that. The fact of the matter is that search engines don't really search the entire internet. They only have a small portion of the internet mapped out, and what they're mapping out are the popular pages, the content that people want to find. So it's a lot easier to find content that's newsworthy or popular than it is to find those kind of deep hidden names and, and sources of talent that we're really looking for. Why? Because Google and Bing need to show you ads, and if what you're searching for isn't really ad-worthy, then they're not going to show you as many ads and they don't make as much money. Think about it this way: if it's free, you are the product. And in this case, you definitely are the product because search engines are selling your searches to their advertisers, and they're gonna give you the content that makes that more profitable. It's not necessarily gonna be somebody's resume. It's probably gonna be more popular, common content. So there might be some walls that you don't see. These walls sometimes are intentional, and sometimes they're just based on popularity and the bias of a search engine algorithm prioritizing high authority sources or high engagement sources. So what happens is there may even be technical challenges that prevent you from seeing some of that hidden content. You see, a search engine can't click on things. It can't drop down, drop down menu choices or check boxes or type things into a search box. So if the website that you expect to find information on requires you to choose from a menu before showing you the information, that information is just not going to be accessible to the search engine. Likewise, if the content is behind some sort of heat map that you have to click on, like choose your country or click on the screen to choose a particular category, these search engines are not going to go around clicking on things. They can only follow plainly obvious links, hyperlinks in these pages. So those are the technical challenges, but there's also other challenges. Like for example, some companies simply don't want search engines to actually go through their content and they set up rules on their servers to specifically tell the search engine not to index their content. As in kind of like, you know, go away, Google. We don't want you to see this stuff. Not because it's secret or because they are trying to hide it, but because they just don't want the search engine hammering their server, looking at their website all the time. And it's really not designed for that kind of consumption. So there's a few reasons why when you go on Google, you don't find all the content that you're expecting. So what do you think you can do about that? And how do you find the hidden pools that might be buried behind these, uh, invisible walls or these technical obstacles? Well, you have a few options. One of them is to make sure that when you're conducting a search, you have enough criteria in your search to where your results are less than 300. It's a little-known fact, but Google, for example, won't show you more than 300 results, and that's essentially 30 pages of 10 results each. Come to think of it, a lot of people don't even go past the first or second page of results. So really, your search needs to come back with very specific results, and anything above 300, you're just never going to get to. So that's one thing you can do. Another thing you can do is to use specific syntax or what's called field search syntax to tell the search engine to look in a very specific part of a page, like the address of the page or the words that are in the name of the title of the page. And then you know that you're drilling down into that content that you want to find and you're not being led along by what the search engine wants you to see. And another possibility is to simply go outside of the mainstream search engines and use other search engines that do index some of that hidden content. You might not be aware of this, but Google actually will only read the first 250 lines of text on any document. Can you imagine some of those bigger resume books and PDF files that have dozens of pages? Google's not even going to get past the very beginning of that. Bing, Microsoft search engine, in contrast, will read the first 500 lines of text. So that's one of the reasons why people will tell you Bing goes deeper and Google goes wider, just because Google will typically find more pages but read less of them, whereas Bing is kind of the opposite. But every search engine is going to have its preferences. So use more than one search engine and make sure that your queries are precise enough that you're getting exactly what you want and not just being served some popular content. So folks, I want to talk about something that affects all of us that use the internet, and that is that search engines and search algorithms have their own agenda. They're really not neutral. In fact, they— the way that they measure results and rank results is purely shaped by profit motives. In other words, they're trying to sell you advertising. As a matter of fact, you are the product. What they actually are selling is your eyeballs for their advertisers. So are, are we really able to trust what search engines say? How much of what we find is actually dictated by what a search engine wants us to see versus what actually exists? Let's talk about that. The fact of the matter is that search engines don't really search the entire internet. They only have a small portion of the internet mapped out, and what they're mapping out are the popular pages, the content that people want to find. So it's a lot easier to find content that's newsworthy or popular than it is to find those kind of deep hidden names and, and sources of talent that we're really looking for. Why? Because Google and Bing need to show you ads. And if what you're searching for isn't really ad-worthy, then they're not going to show you as many ads and they don't make as much money. Think about it this way: if it's free, you are the product. And in this case, you definitely are the product because search engines are selling your searches to their advertisers, and they're going to give you the content that makes that more profitable? It's not necessarily going to be somebody's resume. It's probably going to be more popular common content. So there might be some walls that you don't see. These walls sometimes are intentional, and sometimes they're just based on popularity and the bias of a search engine algorithm prioritizing high authority sources or high engagement sources. So what happens is there may even be technical challenges that prevent you from seeing some of that hidden content. You see, a search engine can't click on things. It can't drop down, drop down menu choices or check boxes or type things into a search box. So if the website that you expect to find information on requires you to choose from a menu before showing you the information, that information is just not going to be accessible to the search engine. Likewise, if the content is behind some sort of heat map that you have to click on, like choose your country or click on the screen to choose a particular category, these search engines are not going to go around clicking on things. They can only follow plainly obvious links, hyperlinks in these pages. So those are the technical challenges, but there's also other challenges. Like for example, some companies simply don't want search engines to actually go through their content and they set up rules on their servers to specifically tell the search engine not to index their content. As in kind of like, you know, go away, Google, we don't want you to see this stuff. Not because it's secret or because they are trying to hide it, but because they just don't want the search engine hammering their server, looking at their website all the time, and it's It's really not designed for that kind of consumption. So there's a few reasons why when you go on Google, you don't find all the content that you're expecting. So what do you think you can do about that? And how do you find the hidden pools that might be buried behind these invisible walls or these technical obstacles? Well, you have a few options. One of them is to make sure that when you're conducting a search, you have enough criteria in your search to where you're results are less than 300. It's a little-known fact, but Google, for example, won't show you more than 300 results, and that's essentially 30 pages of 10 results each. Come to think of it, a lot of people don't even go past the first or second page of results. So really, your search needs to come back with very specific results, and anything above 300, you're just never going to get to. So that's one thing you can do. Another thing you can do is to use specific syntax or what's called field search syntax to tell the search engine to look in a very specific part of a page, like the address of the page or the words that are in the name of the title of the page. And then you know that you're drilling down into that content that you want to find, and you're not being led along by what the search engine wants you to see. And another possibility is to simply go outside of the mainstream search engines and use other search engines that do index some of that hidden content. You might not be aware of this, but Google actually will only read the first 250 lines of text on any document. Can you imagine some of those bigger resume books and PDF files that have dozens of pages? Google's not even gonna get past the very beginning of that. Bing, Microsoft search engine, in contrast, will read the first 500 lines of text. So that's one of the reasons why people will tell you Bing goes deeper and Google goes wider, just because Google will typically find more pages but read less of them, whereas Bing is kind of the opposite. But every search engine is gonna have its preferences. So use more than one search engine and make sure that your queries are precise enough that you're getting exactly what you want and not just being served some popular content. So folks, I want to talk about something that affects all of us that use the internet, and that is that search engines and search algorithms have their own agenda. They're really not neutral. In fact, they— the way that they measure results and rank results is purely shaped by profit motives. In other words, they're trying to sell you advertising. As a matter of fact, you are the product. What they actually are selling is your eyeballs for their advertisers. So are, are we really able to trust what search engines say? How much of what we find is actually dictated by what a search engine wants us to see versus what actually exists? Let's talk about that. The fact of the matter is that search engines don't really search the entire internet. They only have a small portion of the internet mapped out, and what they're mapping out are the popular pages, the content that people want to find. So it's a lot easier to find content that's newsworthy or popular than it is to find those kind of deep hidden names and, and sources of talent that we're really looking for. Why? Because Google and Bing need to show you ads. And if what you're searching for isn't really ad-worthy, then they're not going to show you as many ads and they don't make as much money. Think about it this way: if it's free, you are the product. And in this case, you definitely are the product because search engines are selling your searches to their advertisers, and they're going to give you the content that makes that more profitable? It's not necessarily going to be somebody's resume. It's probably going to be more popular common content. So there might be some walls that you don't see. These walls sometimes are intentional, and sometimes they're just based on popularity and the bias of a search engine algorithm prioritizing high authority sources or high engagement sources. So what happens is there may even be technical challenges that prevent you from seeing some of that hidden content. You see, a search engine can't click on things. It can't drop down, drop down menu choices or check boxes or type things into a search box. So if the website that you expect to find information on requires you to choose from a menu before showing you the information, that information is just not going to be accessible to the search engine. Likewise, if the content is behind some sort of heat map that you have to click on, like choose your country or click on the screen to choose a particular category, these search engines are not going to go around clicking on things. They can only follow plainly obvious links, hyperlinks in these pages. So those are the technical challenges, but there's also other challenges. Like for example, some companies simply don't want search engines to actually go through their content and they set up rules on their servers to specifically tell the search engine not to index their content. As in kind of like, you know, go away Google, we don't want you to see this stuff. Not because it's secret or because they are trying to hide it, but because they just don't want the search engine hammering their server looking at their website all the time and it's it's really not designed for that kind of consumption. So there's a few reasons why when you go on Google, you don't find all the content that you're expecting. So what do you think you can do about that? And how do you find the hidden pools that might be buried behind these invisible walls or these technical obstacles? Well, you have a few options. One of them is to make sure that when you're conducting a search, you have enough criteria in your search to where your results are less than 300. It's a little-known fact, but Google, for example, won't show you more than 300 results, and that's essentially 30 pages of 10 results each. Come to think of it, a lot of people don't even go past the first or second page of results. So really, your search needs to come back with very specific results, and anything above 300, you're just never going to get to. So that's one thing you can do. Another thing you can do is to use specific syntax, or what's called field search syntax, to tell the search engine to look in a very specific part of a page, like the address of the page, or the words that are in the name of the title of the page. And then you know that you're drilling down into that content that you want to find, and you're not being led along by what the search engine wants you to see. And another possibility is to simply go outside of the mainstream search engines and use other search engines that do index some of that hidden content. You might not be aware of this, but Google actually will only read the first 250 lines of text on any document. Can you imagine some of those bigger resume books and PDF files that have dozens of pages, Google's not even going to get past the very beginning of that. Bing, Microsoft search engine, in contrast, will read the first 500 lines of text. So that's one of the reasons why people will tell you Bing goes deeper and Google goes wider, just because Google will typically find more pages but read less of them, whereas Bing is kind of the opposite. But every search engine is going to have its preferences. So use more than one search engine and make sure that your queries are precise enough that you're getting exactly what you want and not just being served some