Evaluating Standards-Based Professional Development for Teachers: A Handbook for Practitioners

Evaluating Standards-Based Professional Development for Teachers: A Handbook for Practitioners Michael Puma Jacqueline Raphael The Urban Institute 2001 Prepared for: Prepared by: U.S. Department of Education The Urban Institute Planning and Evaluation Service Education Policy Center Collette Roney, Project Monitor 2100 M Street, NW 400 Maryland Avenue, SW Suite 500 Washington, DC 20202 Washington, DC 20037 i Contents Introduction. Why Use This Handbook? 4 Some Guiding Principles Used in Writing This Handbook 5 Overview of This Handbook 7 Chapter 1. Professional Development and Systemic Standards-Based Reform 9 Components of Standards-Based Reform 9 Linking Professional Development to Student Outcomes 11 What We Know about the Effects of Professional Development 11 Range of Professional Development Approaches 14 Chapter 2. Evaluation: Basic Definitions and Steps 16 What Is Evaluation? 16 What Are the Benefits of Evaluation? 16 What Are the Types of Evaluation? 18 What Are the Steps in an Evaluation? 20 How Can You Address Concerns about Evaluation? 21 Chapter 3. The Evaluation Frame: Understanding the Journey 23 Gathering Information about the Program 23 Exhibit 1: Information To Collect about the Program 25 Exhibit 2: Linking Intermediate and Final Outcomes 27 Understanding the Program: The Ringwood School District Experience 28 Exhibit 3: The Ringwood School District’s Logic Model 30 Chapter 4. Stakeholders: Deciding Who Will Come on the Trip 31 Creating an Evaluation Team 31 Should You Use an Outside Evaluator? 32 Involve Key Stakeholders in the Evaluation 33 Know Your Constraints 34 Creating an Evaluation Team in the Ringwood School District 34 Chapter 5. Evaluation Goals: Determining Your Destination 36 Establishing Evaluation Goals and Objectives 36 Examples of Questions for an Evaluation of Teacher Professional Development 37 Deciding on the Destination: The Ringwood School District Experience 39 Chapter 6. Evaluation Design: Plotting Your Course 43 Choosing a Design Strategy — Things Evaluators Worry About 43 Using Real-World Examples To Understand Evaluation Design 46 Formative or Process Evaluation Design 46 Applying Formative Evaluation to Teacher Professional Development: The Ringwood Experience Continued 50 Types of Summative or Impact Evaluation 52 iiExhibit 4: Types of Impact Evaluation Designs 53 Randomized Experiments of Professional Development 53 Applying Randomized Experiments to Teacher Professional Development: Two Examples 54 Quasi-Experimental Impact Studies 56 Applying Quasi-Experimental Designs to Teacher Professional Development: The Kramer School District Experience 57 Non-Experimental Impact Studies 60 Choosing an Impact Evaluation Design 60 Chapter 7. Data Collection: Getting Started 62 Deciding What Information to Collect 62 Exhibit 5: Linking Research Questions, Measures, and Data Sources 63 To Sample or Not to Sample: That Is the Question 64 Making the Choices: Kramer School District Revisited 69 Exhibit 6: Example of Kramer School District and Teacher Sample with Multiple Treatments 71 Chapter 8. Data Collection: Choosing the Methods 72 Deciding How to Collect the Data 72 Exhibit 7: Methods of Data Collection 74 Consider a Range of Tools 74 Data Collection Methods for Kramer School District 80 Chapter 9. Data Collection: Creating and Using the Tools 81 Constructing Data Collection Instruments 81 Pre-Testing Your Data Collection Instruments 87 Managing Your Data Collection 88 Logistical Planning for Data Collection 89 A Note about Using Data from Existing Records 91 Protecting Human Subjects and Maintaining Confidentiality 91 Data Collection in Kramer School District 92 Chapter 10. Data Analysis: Understanding What Happened 95 Getting Your Data Ready for Analysis 95 Types of Analysis 98 Getting the Story Down: Kramer School District Continued 99 Chapter 11. The Evaluation Report: Telling the Story 103 Interpretation of Results 103 Reporting 103 Exhibit 8: Sections of the Final Report 106 Where Do You Go from Here? 109 Final Thoughts 110 iii Appendix: Guiding Principles for Eval