Putting Standards to the Test: A Design for Evaluating the Systemic Reform of Education

Putting Standards to the Test:A Design for Evaluating theSystemic Reform of EducationMay 30, 2000Mike PumaJacqueline RaphaelKristen OlsonJane HannawaySubmitted to:Policy Studies Associates1718 Connecticut Avenue, NWWashington, DC 20009Prepared for:Prepared by:US Department of EducationUrban InstitutePlanning and Evaluation ServiceEducation Policy CenterCollette Roney, Project Monitor2100 M Street, NW400 Maryland Avenue, SWSuite 500Washington, DC 20202Washington, DC 20037This work was supported by Contract EA9405301 under subcontract from Policy Studies Associates. The Urban InstituteiCHAPTER I: SYSTEMIC REFORM — THE THEORY...........................................I-1Introduction: A Silent Revolution in American Education............................................................................I-1The Seminal Work of Smith & O’Day..................................................................................................................I-3The Systems Approach: What Is It? How Is It Different?..............................................................................I-9Types of Systems...................................................................................................................................................I-10Parts of Systems.....................................................................................................................................................I-11Information Flow In a System..............................................................................................................................I-12K-12 Education as a System...................................................................................................................................I-13A Conceptual Model — A First Cut...................................................................................................................I-15A Second Perspective............................................................................................................................................I-20The Voice of the Critics..........................................................................................................................................I-21Is This Criticism Warranted?...............................................................................................................................I-24Implications for Evaluation...................................................................................................................................I-25The Rest of This Report..........................................................................................................................................I-26CHAPTER II: THE THEORY IN ACTION.................................................................II-1What We Know..........................................................................................................................................................II-2The State of State Policies.......................................................................................................................................II-3A Major National Systemic Reform Effort is Underway................................................................................II-3But Standards and Assessments Vary in a Number of Important Ways......................................................II-4Accountability Policies Lag in Development are Unstable and Vary Across States.................................II-6The Scope of Systemic Reform Policies Varies Across States, as Does the Sequence in Which DifferentElements are Introduced.......................................................................................................................................II-8State Policy Effects....................................................................................................................................................II-9Local Context Effects............................................................................................................................................II-12Implications for Evaluation Design...................................................................................................................II-15CHAPTER III: EFFECTS ON STUDENT LEARNING...........................................III-1Four Studies of Reform and Student Achievement........................................................................................III-2Texas and North Carolina's Gains on the NAEP.............................................................................................III-2The NSF-sponsored State Systemic Initiatives (SSIs)....................................................................................III-4SSI’s and Equity....................................................................................................................................................III-6Reform in High Poverty Schools..........