Assessing the Methodology
|There are four main aspects of the research
methodology: design, sampling, data collection, the data analysis.
If inappropriate methodology is used, or if appropriate methodology is
used poorly, the results of a study could be misleading.
Research design specifies what group(s) for which data will be collected, to which group(s) and when the intervention will occur, and when the data will collected from each group. The strength of a design, and the possible biases inherent in a design, depend on the type of questions being addressed in the research.
These conditions rarely can prevail when testing complex educational or workplace innovations. It is rare that a convincing placebo can be concocted and administered. It is often difficult to prevent some spill-overs of the treatment--whereby those receiving the treatment share it with some friends who are in the control group. It is also difficult to prevent those not receiving the treatment from seeking alternative treatments on their own. It is rare that those administering the innovation can do so without knowing they are using the “treatment” rather than the “placebo.” It is also rare that those measuring the effects are “blind” about who did and did not receive the treatment, although this is sometimes feasible to arrange. This is not to say that experimental designs are a waste of time in answering causal questions in education and worksite research. Sometimes they are the best option, but rarely are they “golden.”
In qualitative research, often the specific questions of interest emerge in the course of the study and thus the design for answering them must also emerge. While the designs described above tend to be explicitly discussed in quantitative research, they can be applicable to qualitative research. For instance, if the main question is what are recent Central American immigrant youth’s perceptions of DC school life, a phenomenologist could intensively study the perceptions several such youth already in one or more DC schools. If the main question is whether Math Explosion software can boost these youths’ math skills, an ethnographer would have a stronger basis for answering the question if he or she intensively studied these youths’ application of math in school and outside, for awhile before the youth start using the software, during the use, and then afterwards.
Sometimes a study involves the entire population of interest, but more often it involves only a small portion of the students, employees, families, schools, communities, or other “units of analysis.” Sampling serves three purposes:
Qualitative research, and some quantitative research, use non-random samples. Non-random samples include quota samples in which the researcher selects participants in proportion to one or more characteristics of the population. Typical case samples are drawn to represent the median characteristics of the population. Critical cases are drawn to represent certain subgroups of the population that are of particular interest. All these have merit.
The sampling done in qualitative research, however, is often problematic for at least two reasons. First, the researcher may consciously or subconsciously draw cases partly for reasons other than the stated one. For instance, an ethnographer investigating suspected adverse effects of state education reforms on minority youth, may select the few classrooms for intense observation not only to be “typical” but also partly because they are known not to be handling the reforms well. Second, qualitative researchers often don’t explain how they selected the people to observe or to interview, and they rarely tell you what portion of those initially selected refused to cooperate. Consequently, it often is difficult to judge the adequacy of sampling in qualitative research.
Quantitative researchers develop most of their questions and hypotheses very specifically before the study, and then find or develop instruments for collecting the data. That gives them opportunity to refine each item, but no opportunity to address new questions that may arise from the early data collection. Qualitative researchers usually start with a qualitative research methodology (such as historiography, ethnography, phenomenology) and often an interpretive paradigm, and then collect data intensively by observation and unstructured interviews. That allows them to use early findings to generate new questions that they examine in the later stages of data collection, but they often have to focus their observations and develop their interview questions on the fly without any opportunity to refine them.
The means of data collection in social science are diverse. For instance, one can observe and code or note, administer tests of skills, administer various personality and attitude inventories, interview people in person or by phone, mail out questionnaires, content-analyze transcripts of dialogue, and review official documents.
There are two key elements of data collection in quantitative research: the instruments and the data collection procedures. The term “instruments” in the social sciences usually refers to written forms on which the researchers or the people being studied record information. Mechanical and electrical measure are also occasionally used.
Two concepts are central to quantitative measurement: reliability and validity. Reliability means the instrument consistently gives the same value when measuring any given level of a phenomena. Validity means that the value yielded by the instrument is accurate. Reliability is necessary but not sufficient for valid measurement. For instance, careful use of a ruler will allow measurements accurate within about 1/16 of an inch, but the measurements will not be accurate if the user unknowingly has a ruler that has shrunk one inch. Some measures in quantitative social science have credible evidence of their reliability and validity, but most do not and thus must be judged on whatever is apparent from reviewing them. Do the instruments seem to cover all the important issues? Is there balance or do most of the items address only strengths or weaknesses? Is a wide range of responses, ratings, scores, etc. possible? Are the instruments easy to use correctly? Were new instruments developed specifically for the study pilot tested? Who collected the data, with what advance training, and what introductions and instructions provided to the participants, and with what monitoring of the data collection?
Qualitative research relies much less on instruments, making the procedures all-important. The data collection is usually done by doctoral students or scholars, rather than delegated to people with lesser research experience-which is often done in quantitative research. Qualitative research reports usually provide a very general idea of how the data was collected but provide few specifics. These reports rarely indicate what questions were posed in the interviews-indeed the questions often vary from one interviewee to the other, making a report of the questions impractical. The reports also rarely indicate what potentially important events were not observed because of various constraints. Often the only practical way to assess qualitative research data collection is to check whether the investigator at least sought data to challenge or verify his or her early results.
Virtually all data collection methods have their shortcomings and potential biases. Experienced researchers, both quantitative and qualitative, know it is best to try to measure the most important variables with multiple items and/or multiple means, and then compare the results.
In quantitative research, well established statistical procedures are usually used. The appropriateness of the selected procedures can be judged by two criteria. The first is whether the design and data meet the assumptions of the procedure. Some of the more common assumptions are in respect to the level of measurement (nominal ordinal, interval and ratio), normality of distributions (for parametric statistics), and homogeneity of variance (for ANOVA). The second criteria is whether the selected statistical procedure is the most powerful of the available procedures whose assumptions were met.
There is an important aspect of quantitative data analysis that is more difficult to judge-the care with which the data were handled before the analysis and the care with which the data analysis was actually conducted. Manually recorded data almost always includes errors. Some of the errors can be identified by reviewing the data forms, and for some of those identified errors, the correct value can be inferred. Data entry into the computer usually results in some errors, and those can be detected by a second independent keying and automatic check, or by visual comparison of the data forms and the computer record. Some additional data errors can be identified by computer edits for values that are out of the eligible range or inconsistent with each other. In addition to data errors, there can be errors in the commands given to the statistical software. The classic warning of professional data processors is “Garbage In, Garbage Out (GIGO).”
The reader of a research report may detect some errors from implausible results or inconsistencies within or between the tabulated results. Otherwise the best assessment of the data handling is to look in the report for an indication that the data were manually edited, the data entry was verified, and the data file was subjected to further computer edits before the analyses began.
The data analysis of qualitative research is generally inductive, interactive, and iterative. It usually involves the identification of categories, themes, relations among both, and the cross verification of tentative answers to descriptive, associational, and causal questions. The analysis is often described or implied in the discussion of the findings. Competent and careful qualitative data analysis is usually indicated by the researcher exhibiting healthy skepticism, drawing on multiple lines of evidence, and testing his or her early findings with subsequent evidence.
8. Are the data collection instruments and procedures likely to have measured all the important characteristics with reasonable accuracy?
9. Are the quantitative analysis procedures appropriate, does the qualitative analysis cross-verify important findings, and does all the data analysis appear to have been done with care?
Last Update: June 29, 2000