The goal of research is to generalize the results, but there are several things that can get in the way.
This lesson examines three threats to external validity: sample characteristics, stimulus characteristics, and multiple-treatment interference.
Fran is a psychologist who wants to study innovative thinking and how people use it to solve problems in the real world. Because she wants to be able to figure out how to stimulate innovative thinking, she comes up with an experiment to do in a lab.Fran wants to know if reading a short list of problem-solving strategies will help more than reading an article about an innovative thinker. So, she gathers a bunch of participants and gives them each a list of problem-solving strategies, has them read the list and then has them solve a puzzle. She times them to see how fast they are at solving the puzzle.Then, Fran gives them an article about a famous innovative thinker, like Steve Jobs.
After they’ve read the article, they have to solve a similar puzzle. Fran times them on that one, too, to see if their time is faster after having read the article.Fran believes that the faster the person solves the puzzle, the stronger they are at problem solving. As a result, she should be able to tell whether the list or the article is better at stimulating problem solving depending on how fast subjects are at solving the puzzle after they’ve been exposed to one of the treatments.Of course, Fran isn’t interested in just what people do in a lab. She’s really interested in how things work in the real world. External validity is the extent to which the results of an experiment can be generalized to the world at large.
There are several problems that psychologists, like Fran, run into when planning experiments. Let’s look at three threats to external validity: sample characteristics, stimulus characteristics and experimental arrangements.
So, Fran is testing how quickly people solve puzzles as an indicator of how much innovative and problem-solving thinking they are doing. But what if the people she picks are just better at solving puzzles than others?Sample characteristics is when the subjects chosen for the experiment interfere with the results.
For example, if Fran only chooses a few people, and they are all puzzle-solving geniuses, then she might not be able to say anything about what normal people will do.The smaller the sample size of an experiment (that is, the fewer the people studied), the less likely it is that they represent the population as a whole. If Fran only has 10 people in her experiment, then it is less likely that the results can translate to the whole population than if she had 10,000 people in her experiment.
Besides sample size, another sample characteristic that can affect external validity is selection bias, which occurs when only certain people are chosen for an experiment. For example, if Fran has a large number of people in her experiment, but she only selects people who are good at puzzles already, then the results won’t reflect that of the normal, non-puzzle-solving population.
Let’s imagine that Fran takes care of all of the sample characteristic problems.
She gets a large number of participants, and they represent the population pretty well because she doesn’t fall victim to selection bias.There’s still another problem that Fran could face in terms of external validity: experimental arrangements, or the way that the experiment is done. One example of an experimental arrangement issue is stimulus characteristics, or issues with the task in the experiment that could affect external validity.For example, remember that Fran is interested in problem solving and innovative thinking in real life. In the world of her experiment, though, she has people solving a puzzle.
This could be a problem with task generalizability. Is solving a puzzle really like problem solving in real life?Perhaps her subjects are using the same types of strategies and thinking skills during her experiment that they would use to solve problems in real life, but perhaps they aren’t. Stimulus characteristics make it hard to say that the task in the experiment is the same as a real life situation.
But even if Fran’s puzzle is the same as the situations that people encounter in the real world, there are other experimental conditions that could pose a problem for Fran’s experiment.
Another example of an experimental arrangement issue is that of multiple-treatment interference, which is when two or more interventions (or treatments) interact with each other.For example, remember that Fran wants to know if the list of strategies will result in a faster puzzle-solving time than reading the article about an innovative thinker. So, she has her subjects read the list of strategies and times them solving the puzzle. Then she has them read the article on the innovative thinker and times them solving the puzzle.
But what if reading the list of strategies actually affects the way that her subjects read the subsequent article? For example, they might be looking for examples of the strategies in the article about the innovative thinker. As a result, they might do better than they would if they had only read the article and not the list of strategies.Essentially, the question to ask when looking for multiple-treatment interference is: Did one of the treatments affect the other? If the answer is yes, then that could be a problem for the results.
External validity is the extent to which the results of a study are generalizable to the world at large. There are several threats that can lower external validity, among them sample characteristics, stimulus characteristics and multiple-treatment interference.
When this lesson is done, you should be able to:
- Recognize the importance of external validity in studies
- Define sample characteristics
- Describe how stimulus characteristics can impact external validity
- Explain multiple-treatment interference as it relates to research studies