by Wendy Vaulton, Senior Researcher, Center for Reading Recovery and Literacy Collaborative
Although there are some red flags that may tip you off to poor scholarship or propaganda disguised as science, sorting the good from the bad when it comes to educational research can be daunting. This topic is important for practitioners and scholars alike. In fact, the most recent issue of Educational Researcher focuses specifically on the question of what should count as high quality education research (http://edr.sagepub.com/content/43/1/7).
So, while the conversation about judging research quality continues, how can you tell if the next study that comes across your desk is worthwhile or belongs in the garbage bin? Let’s start with the easy stuff. Remember those red flags I mentioned? Be very cautious when:
- A researcher uses inflammatory or insulting language
- Sources cited are almost entirely from special interest groups or popular media (e.g. Wikipedia)
- Study limitations and alternate explanations of findings are ignored or sidestepped
- The author uses logical fallacies and spurious arguments to support a point
It gets trickier when errors and bias in research are subtle or technical. Although not a guarantee of every aspect of quality, research that appears in reputable, peer reviewed journals is generally more credible than articles that appear in other sources. Peer review means researchers who know the field critique and assess an article before it is published to make sure it is accurate, the methods are rigorous, and the topic is relevant to the journal or the publisher. Many government reports are also reviewed by a commission of experts.
Typically, the tone of quality social science is circumspect and judicious. Good research points out its own flaws and limitations rather than obscuring them. Desirable practices including drawing conclusions that are supported by data and avoiding grandiosity and hyperbole. In other words, good research values substance over flash, focusing on the steak rather than the sizzle.
I feel pretty confident saying that to date, there has never been a perfect piece of social science. Bias can enter into a study at any point in its development. What research questions are asked and how they are answered depends on how someone thinks about the issue. Good social science does its best to limit and acknowledge those sources of bias rather than pretending they don’t exist.
A number of people have compiled good checklists of questions for judging the quality of research. Some of the questions that frequently appear on these lists include:
- Does the researcher clearly identify a relevant research problem or question?
- Is there a theory or framework that guides the study and analysis?
- Has the author reviewed the relevant literature thoroughly?
- Is the study’s methodology clearly described? This includes the study’s sample selection criteria, data collection instruments and scientific procedures.
- Are the results presented in a well-organized, easy to follow format?
- Are the limitations of the study discussed?
- Do the researcher’s conclusions follow logically from the results?
- Are the implications of the research of an appropriate size and scope given the nature of the study?
- Does the author have a financial interest in the outcomes of the research?
So take these things into consideration then next time you come across a piece of research. Also, for a good example of bad science, take a look at The Dangers of Bread. This hilarious send-up is a great way to see how easily the trappings of science and statistics get abused to prove a point. You can find it at http://www.geoffmetcalf.com/bread.html
Visit our Center for Reading Recovery and Literacy Collaborative Research and Outcomes page at http://www.lesley.edu/center-for-reading-recovery/research-and-outcomes/