fbpx

Folder Freedom of Information Request (FOIA)

This folder contains documents related to 69 Quality Reviews of What Works Clearinghouse (WWC) publications, conducted between 2007 and 2014. The WWC performs Quality Reviews when individuals believe a WWC publication may contain errors or the WWC did not follow their policies and procedures. This information was acquired through multiple Freedom of Information Act (FOIA) requests by the NIFDI Office of Research and Evaluation. The documentation reveals a wide range of concerns, particularly the misinterpretation of study findings.

Documents

pdf 2007-001 Popular

2007-001

This Quality Review addressed concerns about a WWC intervention report on Reading Recovery. The inquirer was concerned about the WWC's ratings of Reading Recovery as well as the exclusion of relevant studies. The WWC determined no revision was necessary.

pdf 2007-002 Popular

2007-002

This Quality Review addressed concerns about the WWC's evaluation of the 100 Book Challenge reading program. The inquirer was concerned about the inclusion of inappropriate studies in review of the program, the exclusion of a relevant study, and incorrect information about the developer, American Reading Company. The WWC revised the review to clarify the timeframe for their initial literature search.

pdf 2007-003 Popular

2007-003

This Quality Review addressed concerns about the WWC's review of Direct Instruction and specifically the reading program Reading Mastery. The inquirer was concerned about the misinterpretation of study findings, the exclusion of relevant studies, and WWC policies and procedures. The WWC determined no revisions were necessary.

pdf 2008-001 Popular

2008-001

This Quality Review addressed concerns about the WWC's intervention report on Phonological Awareness Training plus Letter Knowledge Training. The inquirer was concerned about the msinterpretation of study findings. The report was revised to correct previous conclusions about attrtition levels.

pdf 2008-002 Popular

2008-002

This Quality Review addressed concerns about the WWC's review of the instructional program Opening Eyes to Math. The inquirer was concerned about the classification of the program as a middle school program. The report was revised to correct the classification of the program.

pdf 2008-003 Popular

2008-003

This Quality Review addressed concerns about the WWC review of New Century Education Software. The inquirer was concerned about the exclusion of relevant studies and WWC policies and procedures. The WWC determined no revisions were necessary.

pdf 2008-004 Popular

2008-004

This Quality Review addressed concerns about the WWC's English Language Learners report listing incorrect information about the First-Grade PALS instructional program. The WWC determined it was necessary to revise the report to correct developer and publisher information.

pdf 2008-005 Popular

2008-005

This Quality Review addressed concerns that the WWC misinterpreted study findings and how this resulted in an inaccurate review of the effectiveness of the instructional program I CAN Learn. The inquirer had additional concerns about the inconsistent application of WWC policies and procedures. The WWC determined no revisions were necessary.

pdf 2008-006 Popular

2008-006

This Quality Review addressed concerns about a WWC intervention report on Reading Recovery. The inquirer was concerned about the misinterpretation of study findings and the exclusion of relevant studies from review. The WWC determined no revisions were necessary.

pdf 2008-007 Popular

2008-007

This Quality Review addressed concerns about the WWC review of the article "Direct Instruction in the preschool." The inquirer believed the WWC misinterpreted the study findings. The WWC determined no revisions were necessary.

pdf 2008-008 Popular

This Quality Review addressed concerns about the WWC's conclusions on Direct Instruction and the reading program Reading Mastery. The inquirers were concerned about the misinterpretation of study findings, the exclusion of relevant studies from review, and WWC policies and procedures. The WWC determined no revisions were necessary.

pdf 2008-009 Popular

2008-009

This Quality Review addressed concerns about the WWC quick review of the study "Charter school performace in Los Angeles unified school district: A district and neighborhood matched comparison analysis." The inquirer was concerned about the misinterpetation fo study findings and WWC policies and procedures. The WWC determined no revisions were necessary.

pdf 2008-010 Popular

2008-010

This Quality Review addressed concerns about the WWC beginning reading review of the Success for All program. The inquirer was concerned about the WWC misinterpreting study findings and that affecting the effectiveness rating of the program. The WWC determined it was necessary to revise their review to clarify descriptions of the studies reviewed.

pdf 2008-011 Popular

2008-011

This Quality Review addressed concerns about the WWC beginning reading intervention report on Read Naturally. The inquirer was concerned about the misinterpretation of study findings, the inclusion of studies where a program was not fully implemented, the exclusion of relevant studies, the inappropriate inclusion of studies, and WWC policies and procedures. The WWC determined it was necessary to revise the report to clarify the level of implementation in a study and the description of a study.

pdf 2008-012 Popular

2008-012

This Quality Review addressed concerns about the WWC beginning reading review of the Accelerated Reader/Reading Renaissance instructional program. The inquirer was concerned about the misinterpretation of a study's findings, the inclusion of a study where a program was not fully implemented, and the inappropriate inclusion of a study. The WWC determined no revisions were necessary.

pdf 2008-013 Popular

2008-013

This Quality Review addressed concerns about the WWC review of the study "The impact of two professional development intrventions on early reading instruction and achievement." The inquirer was concerned about the misinterpretation of study findings, the inclusion of a study where a a program was not fully implemented, and the inappropriate inclusion of a study. The WWC determined a revision was necessary to modify the description of a study to clarify the level of implementation.

pdf 2009-001 Popular

2009-001

This Quality Review addressed concerns about the WWC misinterpreting the studies they review. Four examples of this type of error are provided by the inquirer. The inquirer was also concerned about the inclusion of studies where the program was not fully implemented and WWC policies and procedures. The WWC determined one revision was necessary due to an error with the rating of effectiveness.

pdf 2009-002 Popular

2009-002

This Quality Review addressed concerns about the WWC review of the Wilson Reading System in the WWC beginning reading intervention report. The inquirer was concerned about the misinterpretation of study findings and the inclusion of a study where the program was not fully implemented. The WWC determined it was necessary to revise the report to clarify the level of implementation in a study.

pdf 2009-003 Popular

2009-003

This Quality Review addressed concerns about the WWC review of the High/Scope instructional program. The inquirer was concerned about the exclusion of a relevant study, incorrect information about the program, and WWC policies and procedures. The WWC determined it was necessary to revise their review to remove an inaccurate description of research on High/Scope Preschool Key experiences Series, Booklets, and Videos.

pdf 2009-004 Popular

2009-004

This Quality Review addressed concerns about the WWC's review of the instructional program Reading Recovery. The inquirer was concerned about the misinterpretation of study findings, the exclusion of relevant studies from review, the inappropriate inclusion of studies, and WWC policies and procedures. The WWC determined no revisions were necessary.

pdf 2009-005 Popular

2009-005

This Quality Review addressed concerns about the WWC's review of the Doors to Discovery instructional program. The inquirer was concerned about the misinterpretation of study findings, the exclusion of relevant studies from review, and WWC policies and procedures. The WWC determined it was necessary to revise their review to modify their conclusions on the attrition rates of two studies.

pdf 2009-006 Popular

2009-006

This Quality Review addressed concerns about the WWC's review of the instructional program Reading Recovery in the WWC topic report for beginning reading. The inquirer was concerned about the exclusion of a relevant study from the review. The WWC determined no revisions were necessary.

pdf 2009-007 Popular

2009-007

This Quality Review addressed concerns about the WWC's review of the instructional program Reading Recovery. The inquirer was concerned about the misinterpretation of study findings. The WWC determined no revisions were necessary.

pdf 2009-008 Popular

2009-008

This Quality Review addressed concerns about the WWC's review of the study "Impact evaluation of the U.S. Department of Education's student mentoring program." The inquirer was concerned about the misinterpretation of study findings and WWC policies and procedures. The WWC conclusion about this Quality Review was not provided.

pdf 2010-001 Popular

2010-001

This Quality Review addressed concerns about the WWC's review of the study "Achievement effects of four elementary school math curricula: Findings from first graders in 39 schools." The inquirer was concerned about the misinterpretation of study findings. The WWC conclusion about this Quality Review was not provided.

pdf 2010-002 Popular

2010-002

This Quality Review addressed concerns about the WWC quick review of the study "Impacts of comprehensive teacher induction: Results from the first year of a randomized controlled study." The inquirer was concerned about the misinterpretation of study findings. The WWC conclusion about this Quality Review was not provided.

pdf 2010-003 Popular

2010-003

This Quality Review addressed concerns about the WWC's reviews of the Scott Foresman Reading Street and Success Makers instructional programs. The inquirer was concerned about the exclusion of relevant studies from review and incorrect information about a program. The WWC determined it was necessary to revise their review to clarify the cutoff date for the literature search.

pdf 2010-004 Popular

2010-004

This Quality Review addressed concerns about the WWC's review of the instructional program 100 Book Challenge. The inquirer was concerned about the misinterpretation of study findings, the exclusion of relevant studies from review, the inappropriate inclusion of studies, and incorrect informtation about the program. The WWC determined it was necessary to revise their review to clarify the cutoff date for the literature search.

pdf 2010-005 Popular

2010-005

This Quality Review addressed concerns about the WWC quick review of the study "The New York City aspiring principals program: A school-level evaluation." The inquirer was concerned about the misinterpretation of study findings. The WWC conclusion about this Quality Review was not provided.

pdf 2010-006 Popular

2010-006

This Quality Review addressed concerns about the WWC intervention report on the instructional program Cognitive Tutor Algebra I. The inquirer was concerned about the misinterpretation of study findings, the inappropriate inclusion of a study, and WWC policies and procedures. The WWC conclusion about this Quality Review was not provided.

pdf 2010-007 Popular

2010-007

This Quality Review addressed concerns about the WWC quick review of the study "Culture and the interaction of student ethnicity with reward structure in group learning." The inquirer was concerned about the misinterpretation of study findings. The WWC conclusion about this Quality Review was not provided.

pdf 2010-008 Popular

2010-008

This Quality Review addressed concerns about the WWC beginning reading review of the instructional program Success for All. The inquirer was concerned about the exclusion of a relevant study from review. The WWC determined it was necessary to revise the report to note that one study reviewed did use a control group.

pdf 2010-009 Popular

2010-009

This Quality Review addressed concerns about the WWC intervention report on the instructional program Reading Recovery. The inquirer was concerned about the misinterpretation of study findings, the inappropriate inclusion of studies, and WWC policies and procedures. The WWC determined no revisions were necessary.

pdf 2010-010 Popular

2010-010

This Quality Review addressed concerns about the WWC reviews of beginning reading programs and specifically Reading Mastery. The inquirer was concerned about WWC policies and procedures. The WWC detemrined no revisions were necessary.

pdf 2010-011 Popular

2010-011

This Quality Review addressed concerns about the WWC quick review of the study "The role of simplification and information in college decisions: Results from the H&R Block FAFSA experiment." The inquirer was concerned about the misinterpretation of study findings. The WWC determined it was necessary to revise the report, but specific changes are unkown because the WWC did not send their formal response to this issue.

pdf 2010-012 Popular

2010-012

This Quality Review addressed concerns about the WWC quick review of the study "Charter school performance in New York City." The inquirer was concerned about the misinterpretation of study findings. The WWC conclusion about this Quality Review was not provided.

pdf 2010-013 Popular

2010-013

This Quality Review addressed concerns about the WWC review of the reading program Read Naturally. The inquirer was concerned about the misinterpretation of study findings, the inclusion fo studies where the program was not fully implemented, the exclusion of relevant studies from review, the inappropriate inclusion of studies, and WWC policies and procedures. The WWC determined it was necessary to revise 4 reports to more accurately describe the Read Naturally strategy and Read Naturally program. Reports were also revised to clarify the levels of implementation.

pdf 2010-014 Popular

2010-014

This Quality Review addressed concerns about the WWC quick review of the study "Head start impact study: Final report." The inquirer was concerned about the misinterpretation of study findings. The WWC determined it was necessary to revise the report to clarify the attrition rates.

This Quality Review addressed concerns about the WWC quick review of the study "Head start impact study: Final report." The inquirer was concerned about the misinterpretation of study findings. The WWC determined it was necessary to revise the report to clarify the attrition rates.

pdf 2010-015 Popular

2010-015

This Quality Review addressed concerns about the WWC's review of the reading program Reading Recovery. The inquirer was concerned about the misinterpretation of study findings, the inappropriate inclusion of studies, and WWC policies and procedures. The WWC determined no revisions were necessary.

pdf 2010-016 Popular

2010-016

This Quality Review addressed concerns about the WWC quick review of the study "The Milwaukee parental choice program longitudinal educational growth study: Third year report." The inquirer was concerned about the misinterpretation of study findings. The WWC conclusion about this Quality Review was not provided.

pdf 2010-017 Popular

2010-017

This Quality Review addressed concerns about the WWC quick review of the study "Reading and language outcomes of a five-year randomized evaluation of transitional bilingual education." The inquirer was concerned about the misinterpretation of study findings. The WWC determined it was necessary to revise the report to clarify WWC findings.

pdf 2010-018 Popular

2010-018

This Quality Review addressed concerns about the WWC intervention report on the instructional program Advancement Via Individual Determination (AVID). The inquirer was concerned about the misinterpretation of study findings and the classification of the program. The WWC determined it was necessary to revise the report to clarify the description of a study. Specific details are not available because this information was not supplied by the WWC.

pdf 2011-001 Popular

2011-001

This Quality Review addressed concerns about the WWC intervention report on the effect of Reading Mastery on students with learning disabilities. The inquirer was concerned about the misinterpretation of study findings, the exclusion of relevant studies from review, the inappropriate inclusion of studies, and WWC policies and procedures. The WWC determined it was necessary to revise the report tp remove a study from the analysis of Reading Mastery and change the summary judgment.

pdf 2011-002 Popular

2011-002

This Quality Review addressed concerns about the WWC intervention report on the University of Chicago School Mathematics Project (UCSMP). The inquirer was concerned about the misinterpretation of study findings, the exclusion of relevant studies from review, and WWC policies and procedures. The WWC determined no revisions were necessary.

pdf 2012-001 Popular

2012-001

This Quality Review addressed concerns about the WWC reviews of the instructional programs I CAN Learn and Cognitive Algebra I. The inquirer was concerned about the misinterpretation of study findings, the exclusion of relevant studies from review, and WWC policies and procedures. The WWC determined it was necessary to revise the report to include additional data from a study previously not reported.

pdf 2012-002 Popular

2012-002

This Quality Review addressed concerns about the WWC's reviews of the instructional program Read Naturally. The inquirer was concerned about the misinterpretation of study findings, the inclusion of studies where the program was not fully implemented, and the inappropriate inclusion of studies. The WWC determined it was necessary to revise the intervention report on beginning reading to clarify the level of implementation in one study.

pdf 2012-003 Popular

2012-003

This Quality Review addressed concerns about the WWC's reviews of the instructional program Read Naturally. The inquirer was concerned about the misinterpretation of study findings, the inclusion of studies where the program was not fully implemented, and the inappropriate inclusion of studies. The WWC determined it was necessary to revise the intervention report on beginning reading to clarify the level of implementation in one study.

pdf 2012-004 Popular

2012-004

This Quality Review addressed concerns about the WWC's reviews of the instructional program Read Naturally. The inquirer was concerned about the misinterpretation of study findings, the inclusion of studies where the program was not fully implemented, and the inappropriate inclusion of studies. The WWC determined it was necessary to revise the intervention report on beginning reading to clarify the level of implementation in one study.

pdf 2012-005 Popular

2012-005

This Quality Review addressed concerns about the WWC's reviews of the instructional program Read Naturally. The inquirer was concerned about the misinterpretation of study findings, the inclusion of studies where the program was not fully implemented, and the inappropriate inclusion of studies. The WWC determined it was necessary to revise one report to clarify the level of implementation in one study.

pdf 2012-006 Popular

2012-006

This Quality Review addressed concerns about the WWC's reviews of the instructional program Read Naturally. The inquirer was concerned about the misinterpretation of study findings, the inclusion of studies where the program was not fully implemented, and the inappropriate inclusion of studies. The WWC determined it was necessary to revise one report to clarify the level of implementation in one study.

pdf 2012-007 Popular

2012-007

This Quality Review addressed concerns about the WWC's reviews of the instructional program Read Naturally. The inquirer was concerned about the misinterpretation of study findings, the inclusion of studies where the program was not fully implemented, the exclusion of relevant studies from review, and the inappropriate inclusion of studies. The WWC determined it was necessary to revise one report to clarify the level of implementation in one study.

pdf 2012-008 Popular

2012-008

This Quality Review addressed concerns about the WWC's reviews of the instructional program Read Naturally. The inquirer was concerned about the misinterpretation of study findings, the inclusion of studies where the program was not fully implemented, the exclusion of relevant studies from review, and the inappropriate inclusion of studies. The WWC determined it was necessary to revise one report to clarify the level of implementation in one study.

pdf 2012-009 Popular

2012-009

This Quality Review addressed concerns about the WWC's reviews of the instructional program Read Naturally. The inquirer was concerned about the misinterpretation of study findings, the inclusion of studies where the program was not fully implemented, and the inappropriate inclusion of studies. The WWC determined it was necessary to revise one report to clarify the level of implementation in one study.

pdf 2012-010 Popular

2012-010

This Quality Review addressed concerns about the WWC's reviews of the instructional program Read Naturally. The inquirer was concerned about the misinterpretation of study findings, the inclusion of studies where the program was not fully implemented, and the inappropriate inclusion of studies. The WWC determined it was necessary to revise one report to clarify the level of implementation in one study.

pdf 2012-011 Popular

2012-011

This Quality Review addressed concerns about the WWC's single study review of the study "Effects of problem based economics on high school economics instruction." The inquirer was concerned with the misinterpretation of study findings and WWC policies and procedures. The WWC determined it was necessary to revise the report to correct errors about study findings and clarify the use of WWC procedures in their analysis.

pdf 2012-012 Popular

This Quality Review addressed concerns about the WWC's reviews of Direct Instruction and the reading program Reading Mastery. The inquirer was concerned about the misinterpretation of study findings, the exclusion of relevant studies from review, the inappropriate inclusion of studies, and WWC policies and procedures. The WWC determined it was necessary to revise the report to remove a study from the analysis of Reading Mastery and change the summary judgment.

pdf 2012-013 Popular

2012-013

This Quality Review addressed concerns about the WWC's review of the study "A little now for a lot later: A look at a Texas advanced placement incentive program." The inquirer was concerned about the misinterpretation of study findings, the exclusion of relevant studies from review, and the inappropriate inclusion of a study. The WWC determined it was necessary to revise their database to include the preliminary and final version of the study.

pdf 2012-014 Popular

2012-014

This Quality Review addressed concerns about the WWC's review of Reading Mastery. The inquirer was concerned about the misinterpretation of study findings, the exclusion of relevant studies from review, the inappropriate inclusion of studies, and WWC policies and procedures. The WWC determined it was necessary to revise their intervention report on the effects of Reading Mastery on students with learning disabilities to remove a study from their analysis and change the summary judgment.

pdf 2012-015 Popular

2012-015

This Quality Review addressed concerns about the WWC's quick review of the study "Reading and language outcomes of a five-year randomized evaluation of transitional bilingual education." The inquirer was concerned about the misinterpretation of study findings. The WWC determined it was necessary to revise the report to provide greater clarity on the study findings.

pdf 2012-016 Popular

2012-016

This Quality Review addressed concerns about the WWC quick review of the study "Charter school performace in New Jersey." The inquirer was concerned about the misinterpretation of study findings. The WWC determined no revision were necessary.

pdf 2013-001 Popular

2013-001

This Quality Review addressed concerns about intervention reports on Reading Mastery and Reading Recovery. The inquirer was concerned about the misinterpretation of study findings, inappropriate inclusion of studies, and WWC policies and procedures. The WWC determined it was necessary to revise the intervention report on Reading Mastery to note the report was revised from a previous version that contained erros. The Reading Recovery intervention report was revised to clarify the study disposition for the Baenen et al. (1997) study.

pdf 2013-002 Popular

2013-002

This Quality Review addressed concerns about the WWC's review of the instructional program Read Naturally. The inquirer was concerned about the misinterpretation of study findings. The WWC determined it was necessary to revise the report to clarify the description of Read Naturally.

pdf 2014-3: What is a Valid Scientific Study? An Analysis of Selection Criteria Used by the What Works Clearinghouse Popular

2014-3: What is a Valid Scientific Study? An Analysis of Selection Criteria Used by the What Works Clearinghouse

Meta-analyses and reviews of the educational research literature have identified hundreds of efficacy studies. Yet the What Works Clearinghouse reports that very few of these analyses meet its selection criteria and standards of evidence. This report examines why these differences occur. It finds that the WWC procedures differ markedly from standard practices within the social sciences. The WWC gives no academic or scholarly justification for their policies. Moreover, an empirical, quantitative analysis of the utility of the WWC approach indicates that it provides no “value added” to estimates of a curriculum’s impact. The costs of applying the WWC standards are far from minimal and result in highly selective and potentially biased summaries of the literature. It is suggested that the public would be better served if the WWC adopted the standard methodological practices of the social sciences.

pdf 2014-4: Reading Mastery for Beginning Readers: An Analysis of Errors in a What Works Clearinghouse Report Popular

2014-4: Reading Mastery for Beginning Readers: An Analysis of Errors in a What Works Clearinghouse Report
A November, 2013, report of the What Works Clearinghouse stated that it could find “no studies of Reading Mastery that fall within the scope of the Beginning Reading review protocol [and] meet What Works Clearinghouse (WWC) evidence standards” (WWC, 2013b, p. 1). This NIFDI technical report documents substantial errors in the WWC’s compilation of studies to examine and in the interpretations of individual studies. Effect sizes (Cohen’s d) were computed for results of more than three dozen studies identified by the WWC but rejected for analysis. All of these studies conformed to standard methodological criteria regarding valid research designs and would have been accepted in scholarly reviews of the literature. The average effect size associated with Reading Mastery (RM) was .57. This value is more than twice the .25 level traditionally used to denote educational significance. The results replicate meta-analyses that have found strong evidence of the efficacy of Reading Mastery. Given the high rate of error in this and other WWC reports consumers are advised to consult reviews of studies in the standard research literature rather than WWC summaries.

pdf 2014-5: Examining the Inaccuracies and Mystifying Policies and Standards of the What Works Clearinghouse: Findings from a FOIA Request Popular

2014-5: Examining the Inaccuracies and Mystifying Policies and Standards of the What Works Clearinghouse: Findings from a FOIA Request

Reviewing documentation related to 62 Quality Reviews of What Works Clearinghouse (WWC) publications, this report summarizes the reasons for the reviews, the revisions made and not made, and the inconsistent application of WWC policies during the publication of these reports and during the Quality Reviews. This information was obtained in the fall of 2013 via the Freedom of Information Act (FOIA). These Quality Reviews were performed in response to the concerns of 54 organizations, study authors, program developers, teachers, and education researchers.Forty-one different instructional programs and study reviews were examined in these Quality Reviews. This documentation revealed a wide range of concerns, particularly the misinterpretation of study findings. This issue was given specific attention, especially in relation to how the WWC accounts for the fidelity of implementation when determining their rating of effectiveness for specific programs. With the information provided from the FOIA request and the publicly available information three conclusions appear clear. 1) The WWC suffers from a lack of transparency in their policies and guidelines, 2) the conclusions they create in their reports can be misleading, and 3) the reports are potentially damaging to program developers and ultimately the success of students.

Module-Bottom-Button-A rev

Module-Bottom-Button-B rev

Module-Bottom-Button-C rev2

AmazonSmileModule 01