Chapter 22. Evaluating a research report (p. 541)
General evaluation criteria
1. Introduction
1.1. Problem (p. 542)
- Is there a statement of the problem? Does the problem indicate a particular focus of study?
- Is the problem "researchable"; that is, can it be investigated through collecting and analyzing data?
- Is background information on the problem presented?
- Is the educational significance of the problem discussed?
- Does the problem statement indicate the variables of interest and the specific relationship between those variables that were investigated?
- When necessary, are variables directly or operationally defined?
- Did the researcher have the knowledge and skills to carry out the research?
- Is the review comprehensive?
- Are all cited references relevant to the problem under investigation?
- Are most of the sources primary (i.e., are there only a few or no secondary sources)?
- Have the references been analyzed and critiqued and the results of various studies compared and contrasted? That, is the review more than a series of abstracts or annotations?
- Is the relevancy of each reference explained?
- Is the review well organized? Does it logically flow in such a way that the references least related to the problem are discussed first and lose most related are discussed last? Does it educate the reader about the problem or topic?
- Does the review conclude with a summary and interpretation of the literature and its implications for the problem investigated?
- Do the implications discussed form an empirical or theoretical rationale for the hypotheses that follow?
- Are references cited completely and accurately?
- Are specific research questions listed or specific hypotheses stated?
- Does each hypothesis state an expected relationship or difference?
- If necessary, are variables directly or operationally defined?
- Is each hypothesis testable?
2. Method (p. 543)
2.1. Participants
- Are the size and major characteristics of the population studied?
- Are the accessible and target populations described?
- If a sample was selected, is the method of selecting the sample clearly described?
- Does the method of sample selection suggest any limitations or biases in the sample? For example, was stratified sampling used to obtain sample described?
- If the study is quantitative, does the sample size meet the suggested guidelines for the minimum sample size appropriate for the method of research represented?
- Do instruments and their administration meet guidelines for protecting human subjects? Were needed permissions obtained?
- Is the rationale given for the selection of the instruments (or measurements) used?
- Are the purpose, content, validity, and reliability of each instruments described?
- Are the instruments appropriate for measuring the intended variables?
- Does the researcher have the needed skills or experience to construct or administer an instrument?
- Is evidence presented to indicate that the instruments are appropriate for the intended sample? For example, is the reading level of an instrument suitable for sample participants?
- If appropriate, are subtest reliabilities given?
- If an instrument was developed specifically for the study, are the procedures involved in its development and validation described?
- If an instrument was developed specifically for the study, are administration, scoring or tabulating, and interpretation procedures fully described?
- Was the correct type of instrument used for data collection (or, for example, was a norm-referenced instrument used when a criterion-referenced one was more suitable)?
- Are the design and procedures appropriate for examining the research question or testing the hypotheses of the study?
- Are the procedures described in sufficient detail to permit replication by another researcher?
- Do procedures logically relate to one another?
- Were instruments and procedures applied correctly?
- If a pilot study was conducted, are its execution and results described as well as its effect on the subsequent study?
- Are control procedures described?
- Does the researcher discuss or account for any potentially confounding variable that he or she was unable to control
3. Results
- Are appropriate descriptive statistics presented?
- Was the probability level at which the results of the tests of significance were evaluated specified in advance of the data analyses? Was every hypothesis tested?
- If parametric tests were used, is there evidence that the researcher avoided violating the required assumptions for parametric tests?
- Are the described tests of significance appropriate, given the hypotheses and design of the study?
- Was the inductive logic used to produce results in a qualitative study made explicit?
- Are the tests of significance interpreted using the appropriate degrees of freedom?
- Are the results clearly described?
- Are the tables and figures (if any) well organized and easy to understand?
- Are the data in each table and figure described in the text?
4. Discussion
- Is each result discussed in terms of the original hypothesis or topic to which it relates?
- Is each result discussed in terms of its agreement or disagreement with previous results obtained by other researchers in other studies?
- Are generalizations consistent with the results?
- Are the possible effects of uncontrolled variables on the results discussed?
- Are theoretical and practical implications of the findings discussed?
- Are recommendations for future action made?
- Are the suggestions for future action based on practical significance or on statistical significance only (i.e., has the author avoided confusing practical and statistical significance)?
5. Abstract or Summary
- Is the problem restated?
- Are the number and type of subjects and instruments described?
- Is the design used identified?
- Are procedures described?
- Are the major results and conclusions restated?
Type-specific evaluation criteria
1. Qualitative research (in general)
- Does the topic studied describe a general sense of the study focus?
- Does the researcher state a "guiding hypothesis" for the investigation?
- Is the application of the qualitative method chosen described in detail?
- Is the context of the qualitative study described in detail?
- Is the purposive sampling procedure described and related to the study focus?
- Is each data collection strategy described?
- Is the researcher's role stated (e.g., nonparticipant observer, participant observer, interviewer, etc.)?
- Is the research site and the researcher's entry into it described?
- Were the data collection strategies used appropriately, given the purpose of the study?
- Were strategies used to strengthen the validity and reliability of the data (e.g. triangulation)
- Is there a description of how any unexpected ethical issues were handled?
- Are strategies used to minimize observer bias and observer effect described?
- Are the researcher's reactions and notes differentiated from descriptive field notes?
- Are data coding strategies described and examples of coded data given?
- Is the inductive logic applied to the data to produce results stated in detail?
- Are conclusions supported by data (e.g., are direct quotations from participants used to illustrate points made)?
- Were the interview procedures pretested?
- Are pilot study procedures and results described?
- Does each item in the interview guide relate to a specific objective of the study?
- When necessary, is a point of reference given in the guide for interview items?
- Are leading questions avoided in the interview guide?
- Is the language and complexity of the questions appropriate for the participants?
- Does the interview guide indicate the type and amount of prompting and probing that was permitted?
- Are the qualifications and special training of the interviewers described?
- Is the method used to record responses described?
- Did the researcher use the most reliable, unbiased method of recording response that could have been used?
- Does the researcher specify how the response to semistructured and unstructured items were quantified and analyzed?
- Does the written account (the ethnography) capture the social, cultural, economic themes that emerged from the study?
- Did the researcher spend a "full cycle" in the field studying the phenomenon?
No comments:
Post a Comment