"The 'P-Value' Trap: Why Your Significant Results are Actually Useless"

Statistical versus practical significance. Why significant results may still be meaningless. Your results can be statistically significant and still relatively meaningless in the real world. In this video, I'm going to walk you through 3 steps where we look at the differences. This will allow you to not overclaim things with your dissertation or rely on what many people might think of as flashy statistics. I'm Dr. Steve Tippins Ph.D. I've spent over 35 years as an academic, served on over 300 dissertation committees, and helped hundreds of doctoral students get their research and dissertations accepted. In this video, I'll walk you through the questions you need to ask so that your results will be meaningful and also helpful in the real world. So why is this important? In your dissertation, you're not just expected to report a p-value, but also talk about it and why it's important. You can end up with a huge sample and tiny things can be statistically significant, but you need to be able to understand and explain why the results you have are actually meaningful. Now, just as an aside, the word significant has a strong meaning within data analysis, meaning you've met a bar, a p-value, and that means something is significant. However, we use the word significant in the world in many different ways. I want to caution you, especially if you're doing qualitative research, about saying a result is significant because that can imply to a committee member that you're talking about statistical significance and can get to be confusing. So if you can just avoid that word, it can save a few issues potentially moving forward. So what really is statistical significance? Ultimately, we're looking to say, is there evidence to say that something that we're measuring is important at a level that we have said differentiates importance from not important? Statistical significance answers the question, do the results have an answer that I should talk about? Statistical significance is a bar, and if your answer is above or below, depending how you look at it, this bar, it qualifies as significant. If it does not meet that level, or a p-value, or 0.05, which is very common, 0.05 or lower, it's significant. If it's not at that level or a higher number, it is not what is known as statistically significant. All of this depends upon probability and sampling to come up with this answer. Possibly said less confusingly, if something is statistically significant, it's less likely to just be random noise or to have occurred randomly. It means, oh, there's something there. It's not, hey, if a baseball team from the American League wins the World Series, then stock prices go up. There's no real reason to believe anything like that is correlated or is significant. But perhaps we can have something called spurious correlation, and sometimes people report things based upon, oh, the numbers work, but is it actually meaningful? So expanding on that, correlation means things move in a similar pattern. It doesn't mean causation. So the example I used, if an American League team wins the World Series, there's no reason to expect that would have any implication or causation with stock prices. So yeah, correlation, but not causation. Another example could be people have looked at sunspots and stock prices. And stock prices are looked at a lot because there's a lot of data on this and it's easily looked at. So if I can get another set of data and look at it and find some sort of correlation, maybe I can have a leg up on investors, other investors, to see if, oh well, I know that these sunspots have increased, therefore I'm gonna buy stock in a certain company because it goes up. Well, over time, many of these correlations don't pan out as causational. Causational? I think I just made up a word. Here's another example. The number of people who have drowned by falling into a swimming pool correlates to the films that Nicolas Cage has been in. I think it would be hard to bring up causation there. Another one, ice cream sales are correlated with murder rates. Potentially, I could see ice cream sales going down and murder. If I'm not getting ice cream, I might be angry. But ultimately, there's more ice cream sales in the summer, there's more murders in the summer. It's not the ice cream sales that have any causational impact on murders. However, we could say potentially, wow, if the murder rate goes up, many people are going to say, screw the diet, I'm going to eat if I'm going to be murdered, so I'm going to go for ice cream. But again, no real basis for any of this. Practical significance is what your committee is really interested in. Is the size of the effect big enough to really matter in life or in financial matters or in performance matters or in whatever we are measuring. So we could have a difference that turns out to be real, but it's trivial. I've seen students, undergraduate students in particular, report, oh, well, 12.5% of the people believe this, and it was 1 out of 8. So we're taking 1 and saying, yes, 1/8 is 12.5%. But when we bring it into the real world and say 12.5% of people and try to extrapolate out, you know, we're implying levels of accuracy. The other one that always used to bother me was when students were talking about dollars and use their calculator in order to come up with an answer and they come up with 27.915672. And I always answered, when you can provide a level of currency smaller than a penny, you can have your answers go out to more than 2 spots. Because you're implying levels of accuracy that you don't have. Getting out to statistical analysis, many times people are using Likert scales, strongly disagree to strongly agree, and it can be a 1, 2, 3, 4, or 5 or something like that. And then they take these answers of all the 700 or 200 or whatever number of participants there were, and they come up with an average saying, Oh, it's 4.172. Well, nobody was allowed to give an answer other than a 4 or a 5 or a 3 or a 2 or a 1. But that 4.172 implies that there's a lot of accuracy in the data, and you were just taking strongly agrees and turning them into numbers. So you have to be aware of this when you're making claims about what you're reporting. If you have a large sample with little variability, I can still find microscopic events or effects. May not be what you want to be reporting. Your practical significance should be tied to things like cost-benefit, real-world thresholds, stakeholder interests, and is it meaningful change, not just change. I knew a school once that once said, Well, we want to have all of our students talk about social change. And what they meant was social change for good. But they didn't say that when they first started this process. So social change could have been interpreted as, yeah, I want horrible social change. They evolved over time to realize they had done, you know, limited what they were going to get or were going to get things they didn't want. So they now, social change for good became part of their shtick. So how do I evaluate practical significance? I like to use this 3-question test. How big is the effect? And by effect, I mean, what am I measuring for? Am I measuring the number of cavities people have or, and how big a cavity is important? Because there can be cavities in people's teeth, that are important. They just watch for a while versus, you know, how big does it get to be an effect or something I'm going to look at. Now I say, compared to what? What is the baseline? If you look at advertising, is really good at making claims based on statistics. And if you look way back 60, 70 years, you would get ads like, 9 out of 10 dentists agree that smoking Lucky Strike or whatever cigarette they were promoting is beneficial for you. And 9 out of 10, what dentists and where'd they get them from and who was sampled? And did I exclude 700 other dentists to get these 9 out of 10 dentists to get there? So what's my baseline? What am I comparing things to? And ultimately, would anybody care about these results? Is it enough time? Is it enough money? Is it enough changes in people's life to make it actually a meaningful change? Getting back to marketing, we measure things all the time and then we report them. Oh, this brand of car is much more reliable than this other brand of car because we've looked at all these tests and looked at how many people have complained and that it's more important. There's a measure in the insurance industry by state insurance departments keep— oh my gosh, this is a complaint ratio, number of complaints they get. And you can look and see that one company has 2,000 complaints and another company has 8 complaints. Okay, yes, 2,000 is higher than 8, but company 1 had 40 million insurance policies in that state. And Company 2 had 18 insurance policies in that state. So is 2,000 really a significant number versus what they've done, as opposed to the 8 out of 18? So be aware of things like this when you're reporting results. In doing statistical research, we use a p-value or alpha of 0.05 as almost godlike. Oh, well, it's 0.05, that means it's significant. But what about 0.051? Or what about 0.049995? Is that really a lot better than 0.050005? But we use a cutoff, an arbitrary cutoff that we've accepted. So just be aware that 0.05 is just something that's been accepted over time. We don't want to report significance without saying, the effect size, you know, what am I measuring? What are my confidence intervals? And if you look at bell curves and how they're spread out, and 95% is within a certain number of standard deviations, well, my interval could be here or my interval could be here for 95%. So be aware again of what we're reporting and what the numbers actually tell us? And then, is there a practical interpretation of this, or am I just reporting to find out that ice cream sales are correlated with murders? So what do committees want in your dissertation? Well, they're going to want to know a p-value, they want to know effect sizes, they want to know confidence intervals. Many times even more important than those reporting is your ability to interpret and to explain what's going on. That's what's really important. Even if you get results that are not significant in the traditional academic sense, you can still report them saying things like, this was not found to be significant in this study. However, future research could be done to explore in different populations or larger sample sizes or using different techniques because there seems to be theoretical support for a relationship here.