Chapter 6 : Validity
Validity
- conceptual and scientific soundness of a research study; accuracy
Internal Validity
- ability to rule-out or make implausible alternate explanations of the results. Demonstrates that IV is responsible for effect
Plausible rival hypothesis- alternative interpretation of the researcher’s hypothesis about the interaction of the independent and dependent variables that provide a reasonable explanation of the findings
Threats to Internal Validity-
History- events that take place during course of study that might have an unintended and uncontrolled-for impact on the study’s final outcome
Maturation- intrinsic changes within participants
Instrumentation- changes in the assessment of independent variable
Testing (practice effect)- effects that taking a test on one occasion may have on subsequent administrations
Statistical Regression- phenomenon where extremely high or low scores tend to revert towards the mean
Selection Biases- systematic differences in the assignment of participants to experimental conditions
Attrition- differential and systematic loss of participants from experimental and control groups
Diffusion or Imitation of Treatment- unintended exposure of control group to actual or similar intervention or when experimental group does not receive intended intervention at all. This can equalize the performance of experimental and control groups
Special Treatment or Reaction of Controls- special treatment to control group may lead participant to react or compensate; therefore equalizing performance
External Validity
- generalizability of the results
Threats to External Validity
Sample characteristics- when results of a study apply to only one particular example. Vary on characteristics such as: age, gender, education, and SES.
Stimulus Characteristics and Settings- environmental phenomenon in which particular features or conditions of the study limit the generalizability
Reactivity of the Experimental Arrangements- participant awareness can impact their attitudes and bx during the course of the study
Multiple Treatment Interference- participants administered more than one experimental intervention within same study or same individual participates in more than one study; research results may be due to the context or series of conditions
Novelty Effects- effect of IV may be due to its uniqueness or novelty and not intervention itself
Reactivity of Assessment- participants awareness that performance is being measured alters performance
Pretest and Posttest Sensitization- effect that pretesting and posttesting may have on bx and responses of participants
Timing of Assessment and Measurement- would the same results been obtained if measurement had occurred at a different time?
Construct Validity
- congruence between the study results and theoretical underpinnings guiding the research. Focus is on IV
Threats relate to unique aspects and design of the study that interfere w/ the researcher’s ability to draw casual inferences from results
Improving Construct Validity
- provide a clear operational definition of the abstract concept or IV
- collect data to demonstrate that the empirical representation of the IV produces the expected outcome
- collect data to show that the empirical representation of the IV does not vary with measures of the related but different conceptual variables
- conduct manipulation checks of the IV
Statistical Validity
- aspects of quantitative evaluation that affect the accuracy of the conclusions drawn from the results of a study
Threats to Statistical Validity
Low statistical power- low probability of detecting the difference between experimental and control conditions even if a difference truly exists
Procedural and participant variability- variability in methodological procedures and a host of participant characteristics, which decreases the likelihood of detecting a difference between the control and experimental conditions
Unreliability of measures- consistency of measures used, unreliable measures introduce more random variability into research design
Multiple comparisons and error rates- as the number of statistical analyses increases, so does the likelihood of finding a significant difference between the experimental and control conditions purely by chance
