Assessing the Results of the Study
|There are five key characteristics of research
results: direction, magnitude, variance (of averages and summaries), statistical
significance (when hypothesis testing), and consistency within the study.
The importance and implications of the results depends on all five.
While the importance of the direction and magnitude of results is obvious, these characteristic sometimes go unreported. There are thousands of correlations reported in the research literature without an indication of whether they were positive or negative correlations. There are thousands of experimental differences reported as statistically significant without an indication of the magnitude of the difference. Qualitative research is usually reported in more detail so direction rarely goes unreported. Qualitative analysis, however, often provides ambiguous indications of magnitude. If it says, “the entry-level employees were initially delighted by the firm’s offer to provide tuition assistance at the local community college” does that mean all, most or many of the entry level employees? And if the qualifier “most” is added to the statement, does that mean at least 90 percent, 80 percent, or 70 percent?
Averages or summaries simplify complexity and that is often helpful, but they can also hide important information. It is very easy to accompany mean values with the variance, and that adds considerable information. Some qualitative researchers are careful to report observed variations, but others, in their effort to indicate themes and patterns, don’t provide a sense of the variation.
While statistical significance is usually reported in quantitative research, it is widely misinterpreted by researchers and by readers. When the samples are small (less than 100), a failure to find statistical significance may mean there is no difference in the population or it may mean that there is a modest difference that was not inferred because of the inadequate power of the test. The power of all statistical tests is positively affected by sample size and inversely affected by the variance in the samples. Conversely, a finding of statistical significance from large samples (greater than 1,000) can result from trivial differences. Furthermore, it should also be noted that when 100 hypotheses are tested at the .05 level of significance, there is a good chance of finding about 5 statistically significant results by chance when there are no real differences in the population.
Most quantitative research involves multiple results, and the pattern of the results is far more important than any one by itself. For that reason, it is important that the report indicate all the results, but that may not be the case. Researchers will usually report all their statistically significant results, but they sometimes fail to report some or all of the results that were not significant. The following example will illustrate how this can be grossly misleading. In a study of the effects of Math Explosion software on Hispanic youths, it might be reported that the intervention had statistically significant results for mathematical computation skills. That seems interesting and encouraging until you learn that four other measures of math achievement did not show statistically significant results! The full set of measures might have included a standardized achievement test with scores for computation, concepts, and problem-solving; the students’ math course grades; and their quantitative score on the SAT college admissions test.
Qualitative researchers don’t have any device comparable to statistical significance. That makes the pattern of their results even more important than in quantitative research. Good qualitative research explicitly cross- verifies important results (“triangulate”) and indicates when some but not all of the evidence points in a given direction.
How can you ascertain whether a research report indicates all the results? Sometimes it will say that is has or has not. Sometimes more measures are discussed in the data collection section than are reported in the results section of the report. In addition, the following three conditions that should raise concerns about selective reporting of the results:
Last Update: June 29, 2000