NIFDI recently released two new technical reports on the procedures used by the What Works Clearinghouse (WWC) and the ways in which they differ from those typically used within the social sciences, as well as errors in the actual process used in review.

What is a Valid Scientific Study? An Analysis of Selection Criteria Used by the What Works Clearinghouse (Technical Report 2014-3)

Meta-analyses and reviews of the educational research literature have identified hundreds of efficacy studies. Yet the What Works Clearinghouse reports that very few of these analyses meet its selection criteria and standards of evidence. This report examines why these differences occur. It finds that the WWC procedures differ markedly from standard practices within the social sciences. The WWC gives no academic or scholarly justification for their policies. Moreover, an empirical, quantitative analysis of the utility of the WWC approach indicates that it provides no “value added” to estimates of a curriculum’s impact. The costs of applying the WWC standards are far from minimal and result in highly selective and potentially biased summaries of the literature. It is suggested that the public would be better served if the WWC adopted the standard methodological practices of the social sciences.

Reading Mastery for Beginning Readers: An Analysis of Errors in a What Works Clearinghouse Report (Technical Report 2014-4)

A November 2013 report of the What Works Clearinghouse stated that it could find “no studies of Reading Mastery that fall within the scope of the Beginning Reading review protocol [and] meet What Works Clearinghouse evidence standards” (WWC, 2013b, p. 1). This technical report documents substantial errors in the WWC’s compilation of studies to examine and in the interpretations of individual studies. Effect sizes (Cohen’s d) were computed for results of more than three dozen studies identified by the WWC but rejected for analysis. All of these studies conformed to standard methodological criteria regarding valid research designs and would have been accepted in scholarly reviews of the literature. The average effect size associated with Reading Mastery was .57. This value is more than twice the .25 level traditionally used to denote educational significance. The results replicate meta-analyses that have found strong evidence of the efficacy of Reading Mastery. Given the high rate of error in this and other WWC reports, consumers are advised to consult reviews of studies in the standard research literature rather than WWC summaries.

Examining the Inaccuracies and Mystifying Policies and Standards of the What Works Clearinghouse: Findings from a FOIA Request (Technical Report 2014-5) - Added 10/14/14

Reviewing documentation related to 62 Quality Reviews of What Works Clearinghouse (WWC) publications, this report summarizes the reasons for the reviews, the revisions made and not made, and the inconsistent application of WWC policies during the publication of these reports and during the Quality Reviews. This information was obtained in the fall of 2013 via the Freedom of Information Act (FOIA). These Quality Reviews were performed in response to the concerns of 54 organizations, study authors, program developers, teachers, and education researchers.Forty-one different instructional programs and study reviews were examined in these Quality Reviews. This documentation revealed a wide range of concerns, particularly the misinterpretation of study findings. This issue was given specific attention, especially in relation to how the WWC accounts for the fidelity of implementation when determining their rating of effectiveness for specific programs. With the information provided from the FOIA request and the publicly available information three conclusions appear clear. 1) The WWC suffers from a lack of transparency in their policies and guidelines, 2) the conclusions they create in their reports can be misleading, and 3) the reports are potentially damaging to program developers and ultimately the success of students.

See also:  
NIFDI's Webpage on the What Works Clearinghouse

Anyone with questions regarding these reports should contact Dr. Jean Stockard at NIFDI's Office of Research. Dr. Stockard can be reached at 877.485.1973 or via email at research@nifdi.org.

The National Institute for Direct Instruction (NIFDI) has partnered with Elgin Children’s Foundation, a publicly funded non-profit organization based in Tennessee, to provide training and support for a literacy initiative for underserved children in rural Southern Appalachian communities. This partnership is designed to sustain an initiative that began four years ago with over 9,000 students, utilizing Direct Instruction, or “DI”.

Direct Instruction is a model for teaching that emphasizes well-developed and carefully planned lessons designed around small learning increments and clearly prescribed teaching tasks. It is based on the theory that clear instruction eliminates misinterpretations and can greatly accelerate learning. Its creators, Siegfried Engelmann and Wesley Becker, and their colleagues have proved that when correctly applied, DI can improve academic performance as well as students’ positive self-image dramatically.

The National Institute for Direct Instruction, an Oregon-based non-profit founded by Engelmann, will support the initiative’s efforts by conducting regional training sessions for school coaches and administrators and providing on-site follow-up to build capacity at the building and district levels. The training and support services are scheduled to begin this August.

Elgin Children’s Foundation was founded in 2003 to serve children in rural Southern Appalachia, spanning Kentucky, Tennessee and Virginia, with the mission of giving children a chance to learn, grow, and maximize future educational opportunities. In addition to supporting literacy efforts in 45 schools across nine school districts, the foundation also provides dental and medical care to underserved children in the area.

Each month, District Administration (DA) Magazine recognizes a select number of books as "Noteworthy Reading". Most recently, DA recognized abook by NIFDI Press, The Science and Success of Engelmann's Direct Instruction, as one of four titles for recommended reading for leaders in education.

District Administration's Lauren Williams writes, "For almost a half-century, educator Siegfried Engelmann has shown how all children can learn if they are taught effectively with his Direct Instruction (DI) curricular programs. Edited by Jean Stockard, the chapters in this book are written by educators from multiple generations and disciplines, bringing a variety of perspectives on Engelmann's impact. The book also considers the potential for wider acceptance of Engelmann's programs and how they can drive achievement in America's schools."

NIFDI's most recent Technical Report, An Analysis of Achievement Scores of Arthur Academy Schools, examines data from the Stanford Achievement Test (SAT) and Oregon Assessment of Knowledge and Skills (OAKS) on the reading and mathematics achievement of Arthur Academy students from six consecutive school years: 2007-2008 through 2012-2013. 

The data indicate that at the start of kindergarten, Arthur students had achievement scores that were similar to or slightly lower than students in the nation as a whole. However, by the end of their kindergarten year, the situation had reversed as the average Arthur student scored much higher than peers in the nation. This high level of achievement persisted, and even increased, through later grades. In all cases, the changes over time were statistically significant, and the percentage of students scoring at high levels was substantially higher than would be expected relative to national norms.

Arthur Academies are a set of six charter elementary schools in the greater Portland, Oregon metropolitan area. All of the Academies use the Direct Instruction programs, Reading Mastery Signature Edition and Connecting Math Concepts.