Mike Caulfield: Verified Methodology for Fighting Misinformation

Here's something remarkable and not in a good way. Disengaged employees cost US companies almost $1.9 trillion a year. That's a lot of zeros. Why? Because when teams lack the right skills, productivity declines, turnover increases, and strategies stagnate. But it doesn't have to be that way. At Project Management Institute, home of the PMP and globally recognized certifications, your people can learn to collaborate, deliver on time, and actually get things the cost of not upskilling way higher than the investment. Project Management Institute. Find the answers@pmi.org.

I'm Guy Kawasaki, and this is Remarkable People. We're on a mission to make you remarkable. We have two ways to help you be remarkable right now. One is our book, think Remarkable 9 Paths to Transform youm Life and Make a Difference. I hope you'll read it. The other path is today's podcast and we have with us a guest named Mike Caulfield. He is a research scientist from the University of Washington. He works at the center for an Informed Public. Basically, Mike is renowned for his SIFT methodology, S, I, F, T. This is a crucial tool in the fight against online misinformation, and it empowers educators and learners to critically assess online content. Let me explain. The acronym sift. S is for stop, I is for Investigate the source, F is for Find Trusted coverage, and T is traced back to the original. In November 2023, Mike Co authored Verified how to Think Straight, Get Duped Less, and Make Better Decisions about what to Believe Online with another remarkable people guest, S.A. weinberg. Mike's dedication to digital literacy has not only earned him the 2017 Merlot Award, but also recognition from top media sources such as the New York Times, npr, and the Wall Street Journal. So let's welcome Mike Caulfield to Remarkable People. We're going to learn how to sift and figure out the truth of what we see and read and hear online. I'm Guy Kawasaki. This is Remarkable People. And here we go. Are you drinking Liquid Death?

No. I mean, Yeah, I guess. Dr. Pepper. Is that liquid death? No, probably.

Death is a brand. I'm not talking about the carcinogens in.

Dr. Yeah, I mean, at some level, probably liquid death.

All right, let's get serious here because we have to save democracy. So first question, how do you verify news stories?

So let me just set up a little frame about what we talk about when we're talking about verifying, because I think sometimes people have a different conception of what you're doing. Generally, we're looking at a situation where Someone has seen something on the Internet that might be a news story, that might be an article, that might be a website, whatever it is, they've seen something on the Internet, they've had a reaction to it. Normally that reaction is one of two things. Either this is absolutely evidence of everything I thought, I believe and I believe this is right and so forth, or it is, oh, this is nonsense, this is foolishness. And so the question becomes, is the thing what you think it is? And this is a question where we focus on the book instead of is it true or false? We accept that you've come to something, you've already had a reaction. By the time you're checking something's already happened, you already have an impression. The question isn't what is this thing? The question is whether your impression was correct or whether your impression was wrong. And so when we talk about verifying news sources, what we're saying here is you see something on the web and you react to it, maybe because you. It's called the Mississippi Ranger and you're like, oh, well, this is a local paper in Mississippi. I hope there's not a paper called Mississippi Ranger. If there is, it's a made up name, like, no, liable, wait, I'm going.

To Delane right now.

I should have written down some fake names I could use. But you see something called the Mississippi Ranger and you're like, oh, this is just a local paper in Mississippi covering an issue of something that happened there. And so the question is, is it right? If that's your impression of it, was your impression correct? And what we suggest on something like that is going to Wikipedia. So if it's a paper, if it's a paper of any size, and actually I worked on a project getting local newspapers on Wikipedia for a while and coordinating that. If it's a paper of any size, it'll have a Wikipedia page. And you go there and if there's a paper of this name, if there's not, it doesn't necessari mean it's not a paper. But you know, you might want to find something else. It might not be your best first, first stop. Alternatively, you might go, you might find that the Mississippi Ranger is one of a set of papers that's run by a political consultant who runs what is something we call a pink slime network. I don't know if you heard this term, but it's a network of a lot of things that look like they're news producing sites, but really they're auto generated out of this stuff. And usually for some sort of propaganda. And it could turn out that's actually being run by political consultant. And so when you say verify a source, part of what we're saying is, okay, well, if I thought I was getting this from a local news source, and that was behind my impression that, oh, this is really useful evidence for what I believe or don't believe, and it turns out, no, actually this is being run by a political consultant, or no, this is just a spam site, or no, no one's ever heard of this, then it's maybe not as useful to me. And the way you do that, the way you get that context, is with a new source. When you're checking the source is to start with Wikipedia. If you can't find something on Wikipedia, type in the name of the source into use something like Google News, see if it comes up in a Google News, and maybe type something like funding, who funds this location, basic sorts of things that would give you some context on that source.

Okay, but let's. I was on the board of trustees of Wikipedia and yeah, nobody believes in Wikipedia more than me. All right, yeah, but what if somebody says in response to you saying, check Wikipedia, oh, anybody can change anything on Wikipedia. Why would you use Wikipedia as your reference when anybody can say anything?

Well, as you know, because a, that's not really true on Wikipedia. That's true of Wikipedia in 2006. I hope I'm not being unfair here, but I was on Wikipedia in 2006. It was true in 2006. You could get on there, could say a lot of things. Those things would not be noticed for long periods of time. So the first thing is Wikipedia in 2023 is not Wikipedia of 2006 or 2008. There's just been a lot of effort on Wikipedia to build various bots, various things that look for things that don't have citation, vandalism, unsourced changes, new users coming in from unidentified IPs that are strangely editing a lot of pages with PR content, that sort of thing. So there's that issue there. It is true on some smaller pages. You can get away with this and that in Wikipedia for a little bit of time. That's. That's not impossible. But in general, if it's a good Wikipedia page, you don't have to trust the Wikipedia page because you're going to come to the Wikipedia page. Anything that is contested, could potentially be contested, is going to have a link, a footnote to it, and you're going to be able to use those links to verify it. And the thing too is it doesn't have to be perfect. Wikipedia doesn't have to have an answer to every single one of these, because this is the big thing. The web is abundant, right? If you came to the Mississippi Ranger and you want. This is the source I want to use. And it turns out you can't find any information on the Mississippi Ranger. It's not like you're out of luck. It's the Internet. You can go and find a source that you can actually find information on. And so it doesn't have to be perfect, because the question is not is this specific source the perfect source for what I want to do? It's is this source sufficient for what I want to do, or should I move on and find something else? And you can move on, find something else where there's a better Wikipedia page.

Do you think the day will ever come when you're asked this question and your answer is check ChatGPT or Claude or Bard or Gemini or anything? You obviously said check Wikipedia. You did not say check LLM.

Yeah, right now I wouldn't say check in LLM. There's a couple reasons for that. They are improving, but a lot of them, the information is out of date. Some of them are, have gotten better, better with that. They tend to do really well with structure. They don't always do as well with sort of granular facts. So there's masters of style and structure, but the granular facts have been a persistent problem. And interestingly, there were some predictions that a lot of that would be ironed out by now. But there's a particular thing in an LLM that, that people may not realize, which is that those algorithms are set to have a little bit of flexibility in them, like a little bit of play. Otherwise you'd always get exactly the prediction, same prediction for every set of words in front. They'd never get that sort of real generative quality. And that little bit of play that you have to put in there so that it can do some of these things is also the thing that is giving you what people call these hallucinations. It's going to be a little tougher to work that out than I think people realize. Because the same amount, the same, the same thing that's giving you some of the sort of appearance of creativity in the LLM, the thing that people associate with the generativity is the same thing on the other side that sometimes going too far and creating these hallucinations. I'm not saying that it'll never work out. The jury's out for the moment. Our recommendation in IT is that people tend to think that LLMs are great tools for novices. We actually think they tend to be better tools for either experts or sometimes you find cases where a person is an expert in one field and has some moderate knowledge in another to do that. But because you're looking at the output and you have to evaluate it, novices can get overwhelmed with what they would need to. With what they would need to check.

So since we're on the topic of Wikipedia and LLMs, do you think that LLMs are an existential risk to Wikipedia? Because people will go to their favorite LLM and just ask a question where they may have gone to Wikipedia before. So I understand that Wikipedia would be one of the best sources for LLMs. But what happens if people don't go to Wikipedia anymore and they just go to LLMs? It's the same threat for search engines.

Yeah, and it's even worse than that because the other worry is what if people that want to stack up, Wikipedia credibility, Wikipedia clout, what if they start just using LLMs to write their parts to the Wikipedia page? And now you're introducing a lot of these errors potentially into wikipedia via the LLMs. So even if you go to Wikipedia, you're getting LLMs. And I know that Wikipedia is working on some ways to. To do some detection and so forth and some policies about what you can and can't do. But yeah, I mean, it's an issue. Are LLMs an existential threat to Wikipedia? I. I think existential threats don't have to be successful. They just have to threaten your existence. You can have an existential threat that turns out not to result in the death of something. And I think in that way. Yeah, I think so. I think there's a future where LLMs could do that. That would be really sad because of course, a lot of the productive capabilities of LLMs comes from a lot of people putting in time and writing things like Wikipedia. It's a little bit of a. I forget what's the opposite of parasitic system? A symbiotic. You can either have a parasitic system or a symbiotic system system. Right. And there's one future in which LLMs are parasitic. They take all the stuff that people have worked on provided value, they suck out that value, they spit it back at the user, they erode the business model for these other things and kind of just suck their host dry. And then there's a symbiotic future, which is a symbiote is like a parasite, but they live in a way that benefits their host organism. And that symbiotic future, I think, could be one where we figure out how to make these things work together, play to their individual strengths. And we teach people, like, when you want to go to one and when you want to go to the other. But I think we need that symbiotic future. And I think part of that symbiotic future is people figuring out when it's best to consult something like ChatGPT and when it's best to consult Wikipedia.

Okay? Because LLM vis a vis search engines. I know I search on Google a whole lot less these days. You know, when I want a question like, how do I add a HP printer to my Macintosh network? Yeah, I used to go to Google for that and get 475,000 links. But now I go to Perplexity, which is the world's stupidest name for an LLM. But I go to one of these things and it gives me the answer, not links.

Right? And this relates to something in our book, which is that what most people are looking for is a summary thing. Like, your average informational need is a summary, because the number of things in which you're not an expert far exceeds the number of things in which you are an expert. And the business model for summary is not great. It hasn't been great for a while. The business model is in making an argument, you take all your facts and maybe you do a little bit of summarization, but you make an argument, you say, this is the way things should be, and you do that. Or, the business model isn't selling people things like this will solve your problem, not, I want to do a summary. There is this problem that AI addresses for some people, which is that people go to the Internet and they quite rightly want a summary of something. And instead of getting a summary, they get a list of a lot of people making arguments for something instead of, I just want to know what the thing is first. Or a lot of people selling something saying, hey, that problem you have, here's your solution. And people get frustrated with it. And so part of what we have to do, I think, is, and this is outside the scope of the book, but we have to come up with a business model for summary to get people the answers they want. That is not. Again, it's not a parasitic business model where the summary is coming from a lot of work that people did, but not necessarily giving back or supporting the people that did the work. We're starting to Veer off into solving the problems of the world. But that's okay. I'm up for it.

In my simplistic world, if I did a search, how do I add a HP printer to my Macintosh network? Yeah, listen, just like on a search engine, if the right column has ads for toner cartridges and HP printers, I'm fine with that. I don't.

Right, right.

So I want you to answer this question because I don't know how to answer it, which is how do you tell if a large language model is making shit up and having hallucinations?

How do I tell? Generally I consult it for something that I have some idea about already and that's that that would be the sorts of things. And very often I'm checking and understanding that I already have when I'm going there. Now that said, you're talking about there's different sorts of knowledge in they're not the same. You're talking, for example, about procedural knowledge. You want to set something up. There's one nice thing about procedural knowledge, which is assuming you're not operating a nuclear power plant, you try the procedure, if it works, and then if it works, great, you know, that's confirmation. If it doesn't, then you find something else. So for a lot of procedural knowledge, assuming you're not working with dangerous chemicals or something like that, Yeah, I could see someone doing that. They want to know how to do this. And as a matter of fact, the classic example of that is LLMs are really good at writing code. And if you're trying to write a bunch of computer code to reorganize files on your drive by date and rename them or something like that, make a copy of that before you do it. The LLM can write that, you can run it, and you have some confirmation that the information you got back was good. The problem comes, and this is where our book fits into. I'm glad you mentioned this because there was another interview I did and someone was really obsessed with why do we even need this stuff. If I'm just looking up how to set up YouTube TV on my computer or something like that. And yeah, like for that set of things, it's not really a book about that, but there's another set of things where you can't directly verify the knowledge that you're given. And that's different when. So someone says, look, the fentanyl deaths in this country are at all time high. We need a federal intervention for this. They show you a chart and maybe that's true in this case, probably is true, they are right. But there's no way for you to directly go out into the world and verify whether that information was true or false. And for that sort of thing, I wouldn't trust an LLM unless you really know the subject. I would, in that case try to find something that was directly written by a human, particularly a human that has reputational stakes. Someone who, if they don't take care with the truth, is likely to pay at least some reputational consequence, because that's how we build up trust, as we know. Look, if this person gets it wrong, at the very least it's an embarrassing next day at work, which is often enough to have people get things right.

Therefore, I could make the argument that the fact that LLMs have hallucinations means that Wikipedia still has a place in this world.

Oh yeah, I would absolutely agree with that. And one of the things about Wikipedia and the way it's structured is editors have stakes. I don't think people understand this, but some people play Wikipedia like a video game in the sense that there's a dashboard there and when their changes get reverted, it's pain, it's painful. The flip side of that is sometimes you get these wars which get very emotional about it. But people that work on Wikipedia have reputational stakes. They have a dashboard that shows how many times they created an article that stayed up, how many times they contributed an edit that stayed there, how many times it was reversed. And these things keep the majority of people in Wikipedia on Track. You know, ChatGPT, like the company. ChatGPT has stakes, but the actual thing producing the thing doesn't have any. No. And that's a big difference.

No, not at all.

Yeah.

Okay, so let's say that I go to a website and it's got this.org domain and I go to the about page and it talks about ending climate change and making America great again. That's not a good phrase. It looks like it's a legitimate.org.edu something.

Yeah.

And so what tricks do people use to make a site look credible? And in your case, it's owned by a political consultant who's trying to foster, you know, anti union voting or something.

Kill the minimum wage or something like that. Yeah, yeah, yeah. So here's what, here's the core of what most people do is we talk in our book about cheap signals and expensive signals. And an expensive signal is like your reputation. Like it takes a lifetime to build a reputation. You're very careful about your reputation. You have a history with people that you can maybe find online over years. If you're a reporter, you can look at the articles you wrote 20 years ago in the Washington Post and the articles you wrote yesterday at the Guardian. So there's reputation, that's an expensive signal. And then there's what we call cheap signals. And cheap signals are any thing that gives the appearance of authority or expertise or being in a position to know that is relatively cheap to get. So a classic example of that is dot org. The cost of getting a dot org is like 1295 on go to Namecheap and get a dot org. But someone might look at that and they might say, oh, it's a nonprofit organization. But so it's a cheap signal. Being a non profit organization and having a bunch of people that talk about your work over time and a bunch of different, that's very expensive. That takes a long time to cultivate. Buying a dot org does not. In a similar way, having a good layout on these sites, there may have been a time where in the 1990s, having a good layout to the site, having a crisp look, at the very least, it meant that you had some money. You hired a web developer who could sling that code, get something up, cut it all up in Photoshop and lay it all out in HTML, Dreamweaver or something. It signaled something. Maybe not always a lot, but it signaled, look, someone believes in these ideas enough to fund it signals something. Nowadays, it signals nothing. I think most people know this, but in case they don't, you can get a website that looks as good as your average newspaper. Just go to WordPress, pick a template, start typing, and you'll get something. In many cases, it looks cleaner than your average newspaper. Because if you're faking a newspaper, you don't have to run dozens of ads. So that's a cheap signal too. And so the people that want to fool you do is they look at all the things that people look at to get a sense of whether something has a good reputation. And then they look at the ones that they can get done in an hour or get done and you know, get done in two minutes. And they do that. And that's what they use to fool you. And so whenever you're looking at something, what we encourage people to do is think about like, how hard would it be to fake that? And does that require getting in a time machine and building 10 years of relationships, you know, or does that involve going to namecheap and buying a domain name? And there's a vast difference between the two things and what we Found in our work is that people made no distinctions between those. As a matter of fact, people tended to overvalue the cheap stuff because it was more immediately apparent they could see it looking at the page. Whereas they tend to undervalue the expensive stuff because you had to go out and you had to say, hey, if this guy is an expert in this, there's probably at least a newspaper article or two that quotes them as an expert. So that sort of stuff took a little more effort, like, just a little bit more effort. But it's so much better evidence than the stuff that's about the surface of the page or the domain name, or whether they have an email address you can mail, or whether there's an avatar picture of a real person who might be a real person, might be an AI person, might be some other person that doesn't know their pictures being used.

And in this scenario, when you land at some organization's homepage, would you also go to Wikipedia and look up that organization?

Yeah, absolutely. Absolutely. In fact, one of the things we found Wikipedia is best for is telling you what an organization is about. And that doesn't mean telling you whether the organization is true or false. It's like a nonsensical idea. Is it organization true or false? It doesn't even necessarily mean is the organization credible or not. It just means, is this the sort of source that I thought it was, that I thought I was getting my stuff from? So, for example, you mentioned some of these advocacy sites. You might go to an advocacy site, and one person might go to an advocacy site. Stop, stop. Minimumwage.com and it says, we're a coalition of restaurant workers just looking to protect our lifestyle with tips. And this bill's gonna be horrible for us. And that one person might go to that and be like, okay, I know they're not restaurant workers. I know this is run by a lobbyist firm, But I'm interested in seeing what the lobby. What arguments the lobbyist firm is advancing. If that's your jam, then great, go wild. I wanna see what a lobbyist organist thinks. I go to a lobbyist organization page, I find out what the lobbyist, maybe they're making a good argument. Maybe it's something I should think about. But yeah, for most people, like, when they come to something, they think this is a research group or this is a community organization. This is a grassroots organization. I should say. Again, I don't know, like, I'm just making names up here, but. So I hope that's not a URL that's in play. So the idea is you come to that page, you think it's one thing. You go to the Wikipedia page and it says, hey, this organization was originally founded by a coalition of the nuclear energy industry and the coal industry. You know, okay, maybe they have something interesting to say. Maybe there's something. I'm not saying their facts are wrong, but it's also maybe not your best first stop for a summary of what our energy future should look like. You might want to go somewhere else.

Okay, so now tell me, do you think that 100 Twitter employees sitting in Austin will have any impact on Twitter.

Slash X. I guess it depends on what impact you're thinking here.

Any impact is a low bar safeguard truth.

Yeah. Twitter has placed its eggs in the community notes basket and this is a way that users can add labels to things and rate them and so forth. Say it's inspired by Wikipedia. There's some elements of it that are reminiscent of that. There's some that are not. They've invested less in their trust and safety team. I just say approach these things with caution on Twitter. X. I've been advising people, to the extent they stay on it, to veer more towards their following tab at this point than they're for you. Because that algorithm to me seems like it's more and more tuned to just promote sensational content of a bunch of people that I never seen before.

Okay. But Mike, honestly, when you read the news that Elon Musk says we're gonna get a hundred people in Austin to address these issues on Twitter, did you or did you not start laughing? This is a yes or no.

Did I start? I sighed. Let's say that I side. Yeah, yeah. I think if you want to do that at scale, you need the fund it at a, at a better level. I think it's complex. I do think it's complex. I do think that even old Twitter never quite had it right. It's a hard thinking about how to do moderation, how to do labeling, how to do context, how to do all these various things. It's a lot more difficult, I think, than people recognize. You're always looking at these competing goods that you're trying to protect and you're trying to do that in this fast paced environment where you're making decisions in the moment. I think from the perspective of our book, I think for the time being, you're a little bit on your own. I hope we come to a future where context is rightly seen as a core competitive advantage and community feature for any information Offering that. We don't see this as something that is an add on, but we see, look, people are coming to this for information. Information has to be contextualized and we should compete by providing the best sort of contextual engine for that information. That means labels, that means a well staffed team, et cetera. But we're not there yet. I don't think people fully understand that.

My solution to this is that by default, a social media's home feed, I. E. The stuff that's flying past you, it should be only the people you have manually followed because at least that way you can control. If I only want to follow the New York Times, Washington Post and npr, I don't want you shoving shit into my feed from Rudy Giuliani and whatever, QAnon and all that. It seems to me that would go a long. I would pay for that service.

I also like a platform called bluesky and it's got this idea of the customized feed and you opt into a default feed, which is more or less what you're saying here. Everybody in that feed you're following and it has a very simple algorithm you can understand. The initial Blue sky algorithm was like people you follow in content that got like 12 likes. 12 was like the magic number for a while. And then, yeah, you could choose other feeds if you want to go a little wider, you could go a little wider. If you want people you don't know who are talking about sports teams that you like, but maybe not specifically with a hashtag, you got something that pulls that all together. And so I do think that thinking about the user experience in that way is probably the future there. But right now, yeah, right now a lot of platforms, it's one feed and on Twitter, increasingly odd feed.

Okay, so next loaded question. Okay, what do you make of Facebook blocking searches on threads about COVID And they're saying that, oh, you can't search for the word Covid because it's going to lead to disinformation. Honestly, my head is exploding. This is Facebook telling me this.

Yeah, I don't think it's good. Obviously, in general you do want people to be able to find the information they need on the platforms that they're on. I think the current policy environment is such a. In the current political environment as such that there are some subjects that are just a headache to these platforms. I look at Facebook decisions like that and what I see is not someone that wants to be like some sort of Orwellian 1984. I see a company that keeps on getting called in Front of Congress half the time by Democrats and half the time by Republicans is worried, has a lot of headache, doesn't actually sell. They're not selling a lot of ads next to Covid information and just would like the headache to go away. But I don't think it's a great solution because, you know, I mean, it's a great solution. If your site was, like, about knitting and a bunch of people were posting about COVID you might just say, look, no more covet stuff on the knitting site. It's a headache. I don't want to deal with it. But if your site's Facebook, that's different. I don't think it's a great solution.

Up next, remarkable people.

You ask an academic going and looking at a new area that is adjacent to theirs, like they're trying to flex into a new area and they're trying to understand, like, what are some of the consensus opinions of this field. They go to Wikipedia sometimes because you're going to get a really clear summary there of what that is.

Become a little more remarkable with each episode of Remarkable People. It's found on Apple podcasts or wherever you listen to your favorite shows. Welcome back to Remarkable People with Guy Kawasaki. Since we brought up the subject of COVID So let me ask you, Mike, let's say that one day you wake up and your ears are ringing. Okay. And never been ringing before. Now they're ringing. So you, Mike Caulfield, where do you go on the Internet to investigate this ringing in your ears?

I probably go to Dr. Google like everyone else. And yeah. Yeah. The first search that you'll do will tell you that just as you suspected, it proves you have cancer. And then you got to think about. You got to think about what you did wrong with that search. So, yeah, this is the thing. You do a search, you get a set of search results back. And one of the things that we really encourage people to do is look at that set of search results and think, is this not. Is this the answer I want? That's not what you want to engage in. But is the set of sites coming back, the sorts of sites I want, and are they talking about the things that. That I actually expected they'd be talking about is, yeah. I do joke that sometimes when you go to Google and search your symptoms, it always seems like you've got some tragic illness at first. And then it turns out maybe you just have swimmers here, but you put in your symptoms. Sure. And you execute that search. But then you look at that page and one of the things you want to do, one of the things that Google has now is these three little vertical dots on each result. And if you're trying to figure out, hey, who on this page might I want to get an answer from, you can click on those dots and you can find out, oh, look, this particular center is a community hospital. This particular site I have no information on, I don't know who they are. This particular site is a well known site that sells supplements, right? And so you can kind of browse and you can kind of figure out where you want to go. Sometimes what you want to do in that case is you do the search, you find something that seems like it's a good source of information, you check on the vertical dots, you go to that site and maybe you do a search on that site. If you're on the Mayo Clinic or something like that, maybe do an internal search on the Mayo Clinic at the point you find a site that you trust. And also we talk about in this book, don't give Google tells. Don't give it like clues of what you want to hear, what you expect to hear, what you're worried about hearing. Try to be very bland with it. So yeah, again, if you type in ears ringing, is it cancer? You're going to get a lot of pages back that tell you, yes, it's cancer. If you type ears ringing, common explanations, you're gonna maybe get something that might be a better first stop.

How about, will turmeric cure the ringing in my ears?

Exactly, exactly. You put those words together like in general, you're more likely to get something back that's going to say that if you wanted to do that again, you can cue Google in these ways. You're just trying to add words that Google has a synonym engine now too. So you don't have to be perfect with this. You're just trying to put words that try to signal to Google the type of answer that you're looking for. So if you wanted to put would this spice cure cancer? And you really wanted to know, you might put something like fact check after it to say, look, I'm not looking for something that says this. I'm looking for something that investigates this, right, that actually checks this. So you're going to use various, we call them bare keywords. Don't get too fancy. There's a whole Google syntax. I would not bother to learn it because my experience with other people has been they forget it. Invest your time thinking about, look, I have my query, what's a word that's going to signal the sort of genre thing I want back. Is it this spice cures cancer, fact check, Is that what you're looking for? Is it this spice cures cancer, something else? Why do people think this, that sort of thing. Come up with a keyword that kind of cues Google to give you this.

I want you to explain this. You're telling me that if I ask a question like that and I add the two words fact check.

Yeah.

I'm going to get a better response?

If you want a fact check, you're going to, you're going to likely get a fact check.

Oh man, this is worth the price of admission. I didn't know that.

And it's not anything like built into Google. It's just the fact that two things. One, when you put in fact check, Google has a synonym engine that it runs things through. So it goes and looks not only for fact check, it looks for things that might be synonyms of fact check and so forth. Comes up with a series of terms that it expands and it sends out there. And if your search has that and fact check, reality check, checking the truth, whatever it is, it goes in and it puts ages to have that.

So, yeah, so what if I said Donald Trump won the election, fact check, what would happen?

You would get a fact check that would say probably, I think, I hope, would say that Donald Trump won the election in 2016 and did not in 2020.

Okay, and. And how much credence do you give Google News as opposed to Google?

I used to like Google News quite a bit better. It was a little more integrated with the main product, which means that you could jump back and forth. Now it's an interesting thing right now I generally find that if you use keywords, these bare keywords in the Google search, like write the subject you want and write. If you want a newspaper article, write like newspaper article. And it goes through, does this whole cinnamon thing. Cinnamon thing. And he says, I find it gives you pretty good results. If it doesn't, then I do say, well, okay, if you're not getting what you want, maybe go to Google News. But Google News right now is this hybrid of a sort of a news reading environment in a search engine and a number of other things. There's some good features in it too. I generally stop at Google first and then I go to Google News if it has worked out.

And what about Google Scholar?

A Google Scholar can be really helpful. There's a lot of criticism of Google Scholar. Some of the ways that it calculates citations and everything aren't as perfect as some of the old more manual ways, certain things in there can be gamed. There's a recent paper out on gaming, gaming Google Scholar through a variety of means. So it's not perfect. Again, like part of it is in this sort of world of, is this what I think it is? If someone says they're a published academic on some subject, going to Google Scholar and saying, hey, did this person write anything? And did anybody cite it? That's pretty good. You can also type in, if you're interested in a journal, type in the name of a journal into Google and type in the words impact factor, which is like a number that people use to measure journal influence. It's not a precise number. It just tells you for every article published, how many times is it cited. If you take all the articles in a journal over time and then you look at how many times that journal was cited, what's that ratio? But yeah, if it has no impact factor, I worry.

And will this impact factor, will it show you that there's now like scientific journal farms where you pay to get published so that you're cited and you can cite something? And will Google Scholar tell you that this is a P.O. box in St. Petersburg that has published 2,000 articles about turmeric and tinnitus?

Yeah, Google Scholar won't tell you that. I know that they're trying to. As you probably know, every online information service is just a history of a war with some form of spam. And Google Scholar is exactly the same way. There's ways to spam that system and get stuff that look like it has more credibility than it might, but that does tend to be at the margins still. And the other pieces, you don't necessarily have to use one method. One of the things that people have misconstrued about science and scientific articles is, oh, you're going to read one article and it's going to give you the answer. And that's what science is like. Oh, there was a scientific article that showed X and you hear this and the news kind of follows the same trend. And that's not really how things work. Like, what actually happens is this article seems demonstrated. This article seems to not. This article pulls together the articles that seem to nod and the articles demonstrated into something called a meta analysis. And the sort of progress of things like this. And people tend to get too caught up on the individual article that proves everything. Rather than like, when I survey this area, what do the bulk of people say? You don't have to agree with the bulk of people I'm well known for disagreeing with the bulk of people many times. But you do got to know, like what the sort of consensus is. You got to know before you go against the consensus, you got to know what it is. And so we do recommend something we call zooming out, which is rather than jump immediately into, oh, I found this article that shows xyz, step back and try to say, okay, what do the bulk of people say? Like, when can I find an article that summarizes what the research has said up to now? Sometimes that place is Wikipedia. Sometimes one of the things that we have found in talking to academics is as much as academics suspect Wikipedia, when they're teaching their students, they think it's a little fishy. You ask an academic going and looking at a new area that is adjacent to theirs, like they're trying to flex into a new area and they're trying to understand, like, what are some of the consensus opinions of this field. They go to Wikipedia sometimes because you're going to get a really clear summary there of what that is.

Okay, last question for you. So I got to tell you, I don't know which way I should ask this because what I'm trying to get at is this. If you're a parent, what would you tell your kids are the best practices for figuring out the truth? But I also could make the case that if you're a kid, what do you tell your parent about how to figure out what's the truth? So just give us a checklist. These are the best practices.

All right, so you want to know who produced your information, where it came from, and you want to know if there's a claim being made, if there's some sort of assertion being made. You want to know what other people that are what we call in the know, that have some sort of more than usual knowledge about the thing. You want to know what they think of that claim. And that's just where you start. Like, you should know where your information came from, you should know what other people have said about that issue. And if you make sure that you're doing those two things when you enter a new information domain, you're going to do better. If you don't do that, what happens is you get pulled down this sort of garden path of just never ending evidence. Some people criticize me when I talk about this as the rabbit hole, but the rabbit hole, it's not even whether the rabbit hole is true or not the sort of conspiracist rabbit hole. It's that if you Find yourself being pulled from piece of evidence to piece of evidence to piece of evidence to piece of evidence, and you're never backing up and you're saying, hey, where did this come from? Who produced it? Number one, and then two, what's the bigger picture here? What do people in the know say about this? What should I know? You end up endlessly doing what I call evidence foraging, but you're not getting any benefit to it. It's almost compulsive. And so what you want to do is you want to think about those two things. You want to select the stuff you're consuming a little more carefully, intentionally. In the, in the second to last chapter in the book, we talk about some of that as critical ignoring. Figuring out like, rather than just drink off the fire hose of the Internet, figure out what you want, figure out what you would think would constitute good evidence. Good sources invest the effort to figure out if that's what you're doing or if you're just sort of running from quick hit to quick hit odd TikTok.

And just for greater practicality and usefulness, when you say, go and find out what people in the know or are saying or what's the sort of common understanding of something, where do you get that?

Yeah, you probably start, this is not the end of it, but you probably start with typing in something like history of Israel, Gaza, summary, and then looking at that page and saying, okay, according to my own standards, which one of these would be the best? And one of the things I want to stress is you might want something that's very dry, you might want something that has a little bit of an activist lean, but knowing what you're looking at, choosing that. And so what's happening there is you're in this sort of TikTok loop where you're in this little Twitter doom scroll and you're taking your agency back, you're stopping and you're saying, okay, where did this come from? If it's not the sort of thing I want when I search Google or go somewhere else, then what do I want? And then usually you're using some form of search to get there, but it's about taking your agency back and asking those important questions. And if you don't get good answers to them deciding to go somewhere else.

Okay, listen, I think your book and the work you're doing literally could help save the world. And I hope every school teaches a course like this. I mean, wow.

Yeah, we do too.

So that's Mike Caulfield. I hope you learned a few things about sifting through what we hear and see and read online to help us determine what to believe and what not to believe. Don't forget to check out his book called Verified, along with Sam Weinberg, another Remarkable People guest. So honored you go seeking the truth. I'm Guy Kawasaki. This is Remarkable People. My thanks to the rest of the Remarkable People team. That would be Shannon Hernandez and Jeff C. Sound designers C. Not sift C. And then there's Madison Nismer, producer and co author of Think Remarkable. That's the book you all should read. Tessa Knysmer is our researcher. And then there is Louise Magana, Alexis Nishimura, and Fallon Yates. We are the Remarkable People team, and we are on a mission to make you remarkable. Until next time. Mahalo and aloha. This is Remarkable People.