Internal and External Validity

In quantitative research, a study is valid if one could draw meaning and inferences from the results based on methodology employed.  The three ways to look at validity is in (1) Content (do we measure what we wanted), (2) Predictive (do we match similar results, can we predict something), and (3) construct (are these hypothetical or real concepts).  This is not to be confused with reliability & consistency.  Thus, Creswell (2013) warns that if we modify an instrument or combine it with others, the validity and reliability of it could change, and in order to use it we must reestablish its validity and reliability.  There are several threats to validity that exist, either internal (history, maturation, regression, selection, mortality, diffusion of treatment, compensatory/resentful demoralization, compensatory rivalry, testing, and instrumentation) or external (interaction of selection and treatment, interaction of setting and treatment, and interaction of history and treatment).

Sample Validity Considerations: The validity issues are and their mitigation plans

Internal Validity Issues:

Hurricane intensities and tracks can vary annually or even decadally.  As time passes during this study for the 2016 and 2017 Atlantic Ocean Basin this study may run into regression issues.  These regression issues threaten the validity of the study in a way that certain types of weather components may not be the only factors that can increase/decrease hurricane forecasting skill from the average.  To mitigate regression issues, the study could mitigate the effect that these storms with an extreme departure from the average forecast skill have on the final results by eliminating them.  Naturally, the extreme departures from the average forecast skill will, with time, slightly impact the mean, but their results are still too valuable to dismiss.  Finding out which weather components impact these extreme departures from the average forecast skill is what drives this project.  Thus, their removal doesn’t seem to fit in this study and defeats the purpose of knowledge discovery.

External Validity Issues: 

The Eastern Pacific, Central Pacific, and Atlantic Ocean Basin have the same underlying dynamics that can create, intensify and influence the path of tropical cyclones.  However, these three basins still behave differently, thus there is an interaction of setting and treatment threats to the validity of these studies results. Results garnered in this study will not allow me to generalize beyond the Atlantic Ocean Basin. The only way to mitigate this threat to validity is to suggest future research to be conducted on each basin separately.

Resources

Internal validity in qualitative studies

Internal validity is determining the accuracy of the findings in qualitative research from the viewpoints of the researcher, participants or reader (Creswell, 2013). There are many validity strategies like: Triangulation of different data sources, member checking, rich thick description of the findings, clarifying any bias, presenting negative or discrepant information, prolong the time in the field, peer debriefing, external auditor to review the project, etc.

Triangulation of different data sources for observational work is an idea where I would examine evidence from multiple sources of data to justify the themes that I create through coding.  Converging themes from multiple sources of data and/or perspectives from participants would add to the validity of the study.  Thus, in order to increase the validity of the thematic codes would be to present the thematic codes from analysis of multiple sources like:

  • Interviews from N number of participants (until data saturation is reached)
  • Observations of the participants
    • Repeated observations will be taken, during multiple different types of shifts, with or without the same participants and during different random days of the week over a one-month period.
    • Observational Goals: Tracking what information is used (type and time stamps, instrumentations, etc.)
    • Observational Goals 2: Through videotaping, I hope to track conversations between participants sharing the same shift. Field notes would contain: “Why the conversation was initiated?”, “What was discussed?”, “Were there decisions made regarding the area of study”, “What is the bodily-based behavior portrayed by the specialists in the discussion?”, and “What was the outcome of that discussion?”
  • Document Analysis

The aforementioned, in particular, will help ensure internal validity in quiet a few studies.

 References: