Thursday, January 31, 2008

Educational research : competencies for analysis and applications (3): Evaluating a report

Gay, L. R., Mills, G. E., & Airasian, P. (2006). Educational research : competencies for analysis and applications (8th Ed). Upper Saddle River, NJ: Pearson Education.

Chapter 22. Evaluating a research report (p. 541)

General evaluation criteria

1. Introduction

1.1. Problem (p. 542)
  • Is there a statement of the problem? Does the problem indicate a particular focus of study?
  • Is the problem "researchable"; that is, can it be investigated through collecting and analyzing data?
  • Is background information on the problem presented?
  • Is the educational significance of the problem discussed?
  • Does the problem statement indicate the variables of interest and the specific relationship between those variables that were investigated?
  • When necessary, are variables directly or operationally defined?
  • Did the researcher have the knowledge and skills to carry out the research?
1.2. Literature Review
  • Is the review comprehensive?
  • Are all cited references relevant to the problem under investigation?
  • Are most of the sources primary (i.e., are there only a few or no secondary sources)?
  • Have the references been analyzed and critiqued and the results of various studies compared and contrasted? That, is the review more than a series of abstracts or annotations?
  • Is the relevancy of each reference explained?
  • Is the review well organized? Does it logically flow in such a way that the references least related to the problem are discussed first and lose most related are discussed last? Does it educate the reader about the problem or topic?
  • Does the review conclude with a summary and interpretation of the literature and its implications for the problem investigated?
  • Do the implications discussed form an empirical or theoretical rationale for the hypotheses that follow?
  • Are references cited completely and accurately?
1.3. Hypotheses
  • Are specific research questions listed or specific hypotheses stated?
  • Does each hypothesis state an expected relationship or difference?
  • If necessary, are variables directly or operationally defined?
  • Is each hypothesis testable?
2. Method
(p. 543)

2.1. Participants
  • Are the size and major characteristics of the population studied?
  • Are the accessible and target populations described?
  • If a sample was selected, is the method of selecting the sample clearly described?
  • Does the method of sample selection suggest any limitations or biases in the sample? For example, was stratified sampling used to obtain sample described?
  • If the study is quantitative, does the sample size meet the suggested guidelines for the minimum sample size appropriate for the method of research represented?
2.2. Instruments
  • Do instruments and their administration meet guidelines for protecting human subjects? Were needed permissions obtained?
  • Is the rationale given for the selection of the instruments (or measurements) used?
  • Are the purpose, content, validity, and reliability of each instruments described?
  • Are the instruments appropriate for measuring the intended variables?
  • Does the researcher have the needed skills or experience to construct or administer an instrument?
  • Is evidence presented to indicate that the instruments are appropriate for the intended sample? For example, is the reading level of an instrument suitable for sample participants?
  • If appropriate, are subtest reliabilities given?
  • If an instrument was developed specifically for the study, are the procedures involved in its development and validation described?
  • If an instrument was developed specifically for the study, are administration, scoring or tabulating, and interpretation procedures fully described?
  • Was the correct type of instrument used for data collection (or, for example, was a norm-referenced instrument used when a criterion-referenced one was more suitable)?
2.3. Design and procedure
  • Are the design and procedures appropriate for examining the research question or testing the hypotheses of the study?
  • Are the procedures described in sufficient detail to permit replication by another researcher?
  • Do procedures logically relate to one another?
  • Were instruments and procedures applied correctly?
  • If a pilot study was conducted, are its execution and results described as well as its effect on the subsequent study?
  • Are control procedures described?
  • Does the researcher discuss or account for any potentially confounding variable that he or she was unable to control
3. Results
  • Are appropriate descriptive statistics presented?
  • Was the probability level at which the results of the tests of significance were evaluated specified in advance of the data analyses? Was every hypothesis tested?
  • If parametric tests were used, is there evidence that the researcher avoided violating the required assumptions for parametric tests?
  • Are the described tests of significance appropriate, given the hypotheses and design of the study?
  • Was the inductive logic used to produce results in a qualitative study made explicit?
  • Are the tests of significance interpreted using the appropriate degrees of freedom?
  • Are the results clearly described?
  • Are the tables and figures (if any) well organized and easy to understand?
  • Are the data in each table and figure described in the text?
4. Discussion
  • Is each result discussed in terms of the original hypothesis or topic to which it relates?
  • Is each result discussed in terms of its agreement or disagreement with previous results obtained by other researchers in other studies?
  • Are generalizations consistent with the results?
  • Are the possible effects of uncontrolled variables on the results discussed?
  • Are theoretical and practical implications of the findings discussed?
  • Are recommendations for future action made?
  • Are the suggestions for future action based on practical significance or on statistical significance only (i.e., has the author avoided confusing practical and statistical significance)?
5. Abstract or Summary
  • Is the problem restated?
  • Are the number and type of subjects and instruments described?
  • Is the design used identified?
  • Are procedures described?
  • Are the major results and conclusions restated?

Type-specific evaluation criteria

1. Qualitative research (in general)
  • Does the topic studied describe a general sense of the study focus?
  • Does the researcher state a "guiding hypothesis" for the investigation?
  • Is the application of the qualitative method chosen described in detail?
  • Is the context of the qualitative study described in detail?
  • Is the purposive sampling procedure described and related to the study focus?
  • Is each data collection strategy described?
  • Is the researcher's role stated (e.g., nonparticipant observer, participant observer, interviewer, etc.)?
  • Is the research site and the researcher's entry into it described?
  • Were the data collection strategies used appropriately, given the purpose of the study?
  • Were strategies used to strengthen the validity and reliability of the data (e.g. triangulation)
  • Is there a description of how any unexpected ethical issues were handled?
  • Are strategies used to minimize observer bias and observer effect described?
  • Are the researcher's reactions and notes differentiated from descriptive field notes?
  • Are data coding strategies described and examples of coded data given?
  • Is the inductive logic applied to the data to produce results stated in detail?
  • Are conclusions supported by data (e.g., are direct quotations from participants used to illustrate points made)?
1.1. Interview studies
  • Were the interview procedures pretested?
  • Are pilot study procedures and results described?
  • Does each item in the interview guide relate to a specific objective of the study?
  • When necessary, is a point of reference given in the guide for interview items?
  • Are leading questions avoided in the interview guide?
  • Is the language and complexity of the questions appropriate for the participants?
  • Does the interview guide indicate the type and amount of prompting and probing that was permitted?
  • Are the qualifications and special training of the interviewers described?
  • Is the method used to record responses described?
  • Did the researcher use the most reliable, unbiased method of recording response that could have been used?
  • Does the researcher specify how the response to semistructured and unstructured items were quantified and analyzed?
2. Ethnographic research
  • Does the written account (the ethnography) capture the social, cultural, economic themes that emerged from the study?
  • Did the researcher spend a "full cycle" in the field studying the phenomenon?

No comments: