The REAL Reason Your Study Results Are Invalid

Let's talk about internal and external validity. Or another way to think of it is how not to ruin your study. Did you actually do it right? Did you have internal and external validity? Here's the hard truth. You can spend months doing this work and end up with results that people just don't trust. When this happens, either external and or internal validity have most likely been ignored. I'm Dr. Steve Tippins. I've been on over 300 dissertation committees and helped countless students finish their dissertation. I'm going to walk you through five simple steps to look out for and to ensure internal and external validity so that you can finish your dissertation with competence. You can have statistical results in a quantitative study, but are they valid? Did you do the study right? Is it transferable to other situations? You need to keep internal and external validity in mind. You, you're creating your plan. If you don't, your committee and future readers are going to look at your results and say, so what? Why do this study? So first, what is validity? Validity can be the difference between having an interesting study and a study that has results that are credible. Validity, quite simply, is how well do your results reflect reality, the reality of what you're looking at. Internal validity. I like to think of, did I set the study up properly versus external validity. Is this generalizable to the world beyond my study? Is that designed in that way? Another way to think of this is when you're talking to your committee, internal validity, they're saying things like, wow, can you do this? Is this study set up properly versus external validity? Is, well, what will the results mean to the outside world? What you don't want is, is to get to the end of your study and have your committee or a member of your committee even say, well, I don't know if these results are valid. Hopefully we can help you set up the design at the beginning so the concern about validity is handled right up front and we never worry about it again. So let's talk about internal and external validity in plain language again. Internal validity, did you do it right? External validity, does it mean anything beyond the study? One of the things to think of in internal validity, and this typically is involved in a quantitative study. Have you controlled for external or extraneous variables or inputs? For example, you have given 8th graders a sports drink and you want to see how they can dunk a basketball. And that's it. You say, okay, I've given these 8th graders a this sports drink and now can they dunk a basketball? Well, first, what's the height of these 8th graders are all of them 5 foot 8? Are all of them 7 foot 2 or is the variation in there? Because if I'm 7 foot 2, I start a lot closer to be able to dunk than the 5 foot 8 person. So it may not be the sport drink. So we want to think about any variables that can impact the outcome which then impacts your results. Another way of saying this is, do your conclusions warrant being said based on the sample you used, or can the sample provide you data that will make it meaningful results for you to Talk about? Our 7 foot 2 students probably didn't need the sport drink to dunk. Maybe the 5 foot 8 students needed the sport drink. But you know, let's think about other confounding factors. Are we talking about a 10 foot rim or are we dunking off of a cement floor versus a trampoline effect floor? Again, maybe the sport drink didn't come into play. Or did I give this drink to 8th graders and then ask them to dunk when they were 24? Maybe they've grown. So we want to control for any factors that can confound your results. Now for external validity, do my results apply to the outside world? Let's go back to our sport drink example. If I gave the Sport drink to 8th graders and I found that they were able to dunk the basketball, can I translate that to the rest of the world saying, hey, the rest of you basketball players, you should be doing this because look what happened to these eighth graders. And if my internal validity has been handled and controlled, then I have a lot better song to sing about translating this out to the outside world. But if again, I've got an 8th grader who's 4 foot 9 and the rest of them are 7 foot 3, I can say 90% of my 8th graders were able to dunk after having this drink. Well, 90% were able to dunk before having the drink too. So, you know, controlling internal validity helps your external validity as well. Now also, it's the sample that I take controls my external validity. So again, if I want to look at dunking and I look at my sample being NBA basketball players, it's probably not a good study to be designed for external validity purposes. Internal validity helps you deal with cause and effect. External validity, transferability beyond what you've done. Let me just quickly give you a story. There was a physicist years ago who published a paper about how his cold fusion experiments. He was the first person to ever do cold fusion, which was going to be eternal creation of energy greater than the input, which was amazing. And then other people tried to do his study and they couldn't replicate it. External validity. So he did it again and he got the same result and other people tried to validate it. He did it again and they couldn't validate it. He did it again, got the same results. Then he went on a trip and went to three different labs and tried to do it at their labs. After three different times he could not do it. So he called back to his lab and said, I'm coming home tomorrow. We're going to have to figure this out because we can do it, but we can't do it anywhere else. And even I can't do it anywhere else. And he got back to the lab the next day and found out that his main research assistant had committed suicide the night before because he had been fudging the results to try to make his career look bad better. So the external validity was not there. Now, hopefully none of you ever get in a pressure situation like that, but there are impacts to external validity. Step three, kind of protecting internal validity. And I like the idea of you're protecting against things, sneaking in the back door and causing issues in quantitative studies. And you hear about this many times in medical studies we've got a control group, meaning that's a group that's not getting any kind of intervention of whatever you're studying. And then we've got an experimental group and they get the intervention. And that means hopefully we've controlled for other confounding factors that can happen even then. Let's say I'm doing a study over a couple of weeks time. I'm collecting data from 100 people and I get 40. And then there's a major incident that happens in the world or a major announcement that happens that somehow applies to my study. And then I collect the next 60 pieces of data that that major external event may have made my validity problematic. So controlling for as many things as possible is what we're looking for in internal validity. If you are having a control group and a treatment or experimental group first you try not to let either group know where they are. Are you in the control group? Are you in the experimental group? You also try to not let them talk to each other about things, oh we did this versus oh, I did that. And it can come your results. But designing your study in that way can help reduce the any impact of validity issues. When you are setting up groups you want to avoid for what they call selection bias, meaning, well, I'm going to put everybody over here in my treatment group. So it looks like I get really good results because I know who they are. The goal of research is not to just get really good results, it's to answer the question. So typically we randomly put people into groups so that we haven't selected in a way to get a result that I want. When you design a study, you're going to go through your IRB and they're going to ask you, are you going to pay respondents? And if you say yes, I'm going to give everybody a new car. That's kind of biasing results. If they know, hey, if I give you the questions you answers you want, you get a new car, that's no. They want your reward to be such that it doesn't influence the results that you get. And that's going to throw off your validity. And this part of researcher bias, trying to get the results you want is part of your design. It should go all the way back to your problem and purpose statements where if you're starting to say, ultimately I know what the results are and this is what should be done, well, then you've got bias and you want to keep that bias out of of your design. So you will continue to have internal validity. You want to think, do the results matter? Meaning did I get the results I expected or did I get the results that are different and does it matter to my study? Because ultimately results are results, period. And that's what you report. Once we've set up a study that is internally valid, how do I make sure that external validity is there? Protecting for external validity, you want to think about things like can this study be replicated? And by that I mean can somebody else at another location with another sample come up with and do the same study, or can they take my research and try it in a different way with a different sample or a different methodology or something? So externally, will someone else be able to replicate what I'm doing? This is really important. If you start saying, well, A is associated to B or X has a strong relationship to Y, you want to make sure you've designed your study so that it can be done again. Again, let's go back to the basketball example. If I've given my sports drink now, people dunk and they try to replicate it and they give it to eighth graders and none of the eighth graders can dunk. And you didn't explain that 90% of my eighth graders were 7 foot 3. But then the replication is difficult. And this goes to chapter three of your dissertation of reporting your Design step by step by step that allows other people to replicate what you've done. For external validity, we want to avoid sampling problems. Again, let's talk about the 8th graders. We don't want all of them to be 7 foot 3. We want to make sure we have a group of people that we are allowed to sample most. There are protected groups like pregnant women, prisoners, minors, things like that. Getting them involved in a study is much more difficult. So replication also becomes more difficult. When we're doing a quantitative study, we want to make sure we have enough data in order to produce results that have credibility. Have you ever flipped a coin four times and gotten heads four times in a row? That's not uncommon that that could happen. And then what's the next person going to guess? Oh, it's going to be tails? No, it's got a 50% chance. But if I've done it 4,000 times and I've got 3,500 heads, then maybe I've got something wrong with my coin. So your sample size should be big enough so that aberrations fall away. The G power test is the preferred test to determine sample size. Where at all possible. Committees and researchers like random sampling, meaning people are randomly chosen as opposed to samples of convenience. Hey, I sampled all of my family members or I sampled, and here's a big one, I sampled all of the employees that report to me. And that's an issue because you have power over the people who are responding. So they may be answering in a way that they think think you want to hear as opposed to objectively. The next thing I want to do is understand the interplay between internal and external validity. No study is going to be perfect. So we control for as much as we can. In design. You need a design that is going to answer your question. You need a design that you can justify that will answer your questions. You need in a quantitative study, statistical analysis, analysis that's appropriate. I once did a study and it sent it to a journal and I looked at the data and doing a T test was not the appropriate test based on the distribution of the data. So I used a non parametric alternative, submitted it to the journal for review and a reviewer wrote back and said, you know, this is a good study, but it didn't do a T test so we can't accept it. Now, design wise, a T test wasn't the appropriate test. Validity wise, a T test wasn't appropriate. But I had to determine do I want to fight the battle of educating this reviewer who I don't know or what's my other alternative? Well, the alternative I came up with was, okay, here's the T test and the appropriate test so he could decide and readers could decide which one they want wanted to look at. But at least I still had the appropriate test for validity purposes. Every study has limitations. Whether it's the sample you've used or the number of people. Whatever it is, you want to honestly let the reader know what your limitations were. This opens up opportunities for future research. In your design chapter or methodology chapter, you want to have a discussion of internal and external validity. You might want to be thinking about it very early. Even if the template for your school puts it later, you want to have it thought out and outlined early so that your design is set up in that way. When you do have limitations, honestly say what they are again so that your committee can understand that and perhaps help you design to eliminate or at least reduce some of the limitations. Remember, doing a dissertation is actually the first time you're doing a full research project on your own, so there may be some limitations. This is a learning process, so dive deep into the learning process and learn as much as you can. So in summary, internal validity. Did you do it right? Did you set it up right? Right. External validity. So what? What does it mean externally? Can it be transferred outside of what I have done now? I haven't mentioned this. Qualitative studies have limited external validity because their samples are small. But it does open up the field for future research. A little bit of time thinking about validity, both internal and external, at the beginning or very near the beginning of your work is going to save you a lot of headaches and a lot of time doing your research. If you want support on your dissertation journey, join our free dissertation help group. You'll get our five step dissertation acceptance program help from veteran committee chairs, live Q and A sessions, and a one on one session to help you move faster. You don't have to do this alone. Click the link below and join thousands of dissertation students who've decided to. They don't want to do it alone. They don't want to be stuck. I'll see you on the inside.