References and Resources
After reading education research and making a judgment about whether the results and conclusions can be trusted, policymakers need to decide whether and how the research should be used to influence education policy. State or local factors, including the cost of implementation, are obvious influences on policy decisions. In addition, the quality, coherence, applicability and educational significance of the research should be considered.
The quality of education research is influenced by whether the research is:
- Valid – High-quality education research studies have conclusions that can be trusted. Research designs match research questions, and data collection and analyses follow accepted technical standards.
- Connected to prior research – High-quality education research studies build on prior research studies and conclusions. Research reports indicate how the studies contribute to the current knowledge base on education.
- Ethical – High-quality education research studies follow established rules of research ethics. Procedures are used to avoid researcher bias.
- Peer reviewed – High-quality education research studies are reviewed by other education researchers before the findings and conclusions are communicated broadly.
The coherence of education research is influenced by whether the research findings:
- Are based on a theory or conceptual framework – A theory provides the rationale for the research design and guides the interpretation of the results. Because theories propose explanations for observations, theory-driven research gives policymakers the reasons behind particular findings on a policy issue.
- Have been replicated – Findings that have been replicated in several studies provide a stronger basis for making policy changes than those from only one study.
- Are part of a body of research – A body of research on an education program or policy provides conclusions about an issue or program from different studies in various settings and with various participants. A body of research is more informative to policymakers than are a few disconnected studies. (For an example of a body of research, see the literature review by Cooper et al., [2000] on summer school.)
An important factor that influences whether an education research study should be used to guide policymaking is the degree to which the findings of the study apply to the situation of interest to the policymaker. Researchers call this the external validity of the research.
One consideration that influences applicability is the comparability of the setting of the research study and the setting of interest. For example, research on a teacher professional development program in urban school districts might not be applicable to a state in which rural schools are the norm, particularly if teacher collaboration between schools is an important feature of the program. The distances between rural schools could make teacher collaboration extremely difficult.
A second consideration is the comparability of the participants. There is a lack of research, for example, on curricula and instruction for students from ethnic minorities. Participants in most education research studies are White, which calls into question whether the results apply to participants from ethnic minority backgrounds.
Yet another example is that many research studies on the effectiveness of education programs and practices do not disaggregate results for low-achieving and/or at-risk students. A program that facilitates learning for average students might not help struggling learners. The No Child Left Behind Act requires that states disaggregate state test results for subgroups of students. This requirement will likely result in more research on what can help low-achieving students meet state standards. (See Barley et al. [2002] for a research synthesis on classroom strategies to assist at-risk students.)
A third consideration is the comparability of the program or treatment. Unless the treatment or program described in the research study is fundamentally similar to that of the situation of interest, there can be no expectation that the results of the treatment in the situation of interest will be similar to those observed in the research study. For example, if the research study involved giving students laptop computers to take home as part of their language arts curriculum, using the same curriculum but without the laptop computers may not have the same effect.
Education practitioners can help policymakers determine whether a research study or group of studies is applicable to a particular local context. Practitioner knowledge, also referred to as professional wisdom, is an important source of information about the realities of classrooms and schools and the influences of local circumstances. If research settings do not match local contexts (e.g., research on urban schools applied to a rural state), then policymakers must determine the likelihood that the same results will be obtained in their schools. Practitioners can be of great assistance in this instance.
An important question for policymakers and practitioners to ask about research is, “What is the educational significance of these findings?” In other words, what difference will it make to education if a policy or practice is changed or adopted based on research results? Without knowing the educational significance of a research finding, it is difficult, if not impossible, to estimate the costs and benefits of policy changes. One indicator of educational significance in a research study is the effect size of a program or practice. (Researchers refer to effect size as the practical significance of a result, in contrast to its statistical significance.)
There are some limitations to effect sizes. Their calculation requires quantitative data. In addition, effect sizes that are reported in individual research studies indicate the educational significance of a program or practice only for the specific participants and settings in that study. In other words, effect sizes might not apply to the local context in which the program or practice is implemented. For example, an effect size for a program designed for elementary students might be lower if the program is implemented with middle school students.
A meta-analysis reports an average effect size across several studies of an education program or practice. For this reason, a meta-analysis is a more informative tool for making determinations about educational significance than a single research study.
In the end, it is a matter of balancing all the criteria of usefulness in a way that reflects the local circumstances involved in a particular policy decision. First, it is necessary to determine if the research is empirical and the researcher's conclusions are valid. Next, policymakers must decide how much weight to give to the other criteria of research usefulness. The costs of policy decisions and potentially harmful effects are factors that should always be considered in addition to the information provided by the research. When there is little or no useful research on an education topic related to a policy decision, and a change is needed or mandated, then policymakers should find ways to fund the necessary research. In the long run, a policy decision that is informed by research might be far less costly than one that is uninformed.
To see how all of the pieces fit together in assessing the usefulness of research, or to assess the utility of a particular research study, consult the "Research Utility Assessment Guide" in the APPLIED QUICK PRIMER. The guide can be downloaded and printed out to serve as an informal score sheet.
Barley, Z., Lauer, P.A., Arens, S.A., Apthorp, H.S., Englert, K.S., Snow, D., and Akiba, M. (2002). Helping at-risk students meet standards: A synthesis of evidence-based classroom practices (REL deliverable #2002-20). Aurora, CO: Mid-continent Research for Education and Learning.
Cooper, H., Charlton, K., Valentine, J.C., and Muhlenbruck, L. (2000). "Making the most of summer school: A meta-analytic and narrative review." Monographs of the Society for Research in Child Development, Serial No. 260, 65(1).
