Diverse Engagement Profiles: Demonstration and Implications of Test Preparation for High-Stakes Exams

Working Paper 2079-07 Diverse Engagement Profiles: Demonstration and Implications of Test Preparation for High-Stakes Exams Abeer A. Alamri. PhD and Edgar Sanchez. PhD ACT.orgooecoACT® The authors thank Jeff Allen, Scott Payne, and Ruitao Liu for their helpful comments and suggestions on earlier drafts of this research. © 2019 by ACT, Inc. All rights reserved.R1732ACT working papers document preliminary research. The papers are intended to promote discussion and feedback before formal publication. The research does not necessarily reflect the views of ACT.Abeer A. Alamri earned her Ph.D. and Master's Degrees in Educational Measurement and Evaluation from the University of South Florida. She has a broad interest in structural equation modeling, multilevel modeling, Bayesian analysis, finite mixture modeling, and cross-cultural research. Edgar SanchEdgar Sanchez, a senior research scientist in the Validity and Efficacy Research department at ACT, works on predictive modeling of student educational outcomes. He is currently focusing on the efficacy of test preparation programs. 1 Diverse Engagement Profiles: Demonstration and Implications of Test Preparation for High-Stakes Exams Abstract Students from different demographic and academic backgrounds use many different types of test preparation products. Understanding variations in test preparation usage can help inform product improvement efforts and can also help guide how test preparation developers advise new users to get the most from their preparation. In this study, we aim to characterize how students enrolled in a test preparation product, ACT® Online Test Prep (AOP), use that product. Following Geiser (2012), Masyn (2013), and Morin (2016), we use latent profile analysis (LPA) to identify students’ engagement profiles using four engagement measures: activity, time, practice ACT score, and percent correct. Data collected from 9,017 students between December 2015 and June 2018 identified five student engagement profiles as Low-Usage/Low-Performance (24.1%), Low-Usage/High-Performance (7.2%), Low-Usage/Moderate-Performance (31.7%), Moderate-Usage/Low-Performance (16.8%), and High-Usage/Moderate-Performance (20.2%). We found that these engagement profiles were distinguished by activity, time, practice ACT score, and percent correct; they were also differentially predicted by gender in one profile. The heterogeneity of engagement profiles is discussed in relation to usage and performance as well as potential future research directions. Word count 183 Keywords: ACT Online Test Prep, AOP, latent profile analysis, LPA, mixture models 2 Introduction High-stakes standardized testing (e.g., ACT® and SAT) can play an important role in helping students attain their academic goals. As a result, students are motivated to improve their test scores to improve their readiness for college, meet admissions requirements, maximize scholarship opportunities, and avoid remediation. The use of these tests includes measuring college and career readiness, college admissions, scholarships, and course placement. In 2017, just over two million high school graduates, approximately 60% of high school graduates, took the ACT with an average ACT Composite score of 21.0 (ACT, 2017a; ACT, 2017b). Of those students, 27% met all four ACT College Readiness Benchmarks. Students can use many different tools to prepare for the ACT test, including resources from ACT, which provide different types of supports and content review. Within the test preparation literature, research has demonstrated the positive impact of test preparation on test scores (e.g., Briggs, 2009; Messick & Jungeblut, 1981; Moore, Sanchez, & San Pedro, 2018; Sanchez & Harnisher, 2018). Some r esearch has studied motivational factors that may affect student preparation (e.g., Appelrouth, Zabrucky, & Moore, 2015), while other research has investigated the causal evidence of the impact of test preparation on scores (Moore et al., 2018). However, different usage patterns of test preparation products have not been studied. Therefore, we seek to identify the emergent profiles of users of ACT Online Prep (AOP). In this manuscript, we elaborate on key features of AOP, present the empirical data, discuss the results, and summarize recommendations for future researchers. 3 ACT Online Prep (AOP) ACT1 offers a variety of test preparation materials that feature real questions and sample materials from previous tests including: the Real ACT Prep Workbook; ACT® Academy™; ACT Online Prep (AOP); ACT Rapid Review; and a collaboration with Kaplan Test Prep to offer an interactive virtual classroom called ACT® Kaplan® Online Prep Live (OPL). In this paper, we limited our examination of test preparation to the AOP product. AOP is a subscription-based online service with access to a dynamic, interactive test preparation course designed by ACT. It consists of six components that target skills in the four subjects (English, mathem