CRASE5用于ACT写作技术报告

Scott W. Wood, Sungjin Nam, and Dongmei Li I. Introduction Starting in October 2022, ACT began using its automated scoring engine, CRASE®, to provideone of the two rater scores for ACT writing essays in the ACT International program. Since then,CRASE has been used for other programs, including ACT District (Spring 2023), ACT State To support this decision, ACT created a significant research agenda. Researchers conductedmultiple proof-of-concept studies to evaluate the accuracy of CRASE scores compared to those Since CRASE began to score ACT writing, there has been a desire to add new enginefunctionalities in order to expand the kinds of essays CRASE can handle. The newfunctionalities include automatically detecting off-topic essays, automatically detecting disturbingcontent, leveraging modern model-fitting approaches, and providing information about the The primary purpose of this report is to replicate the studies listed in the CRASE+ technicalreport using the new generic scoring models produced in CRASE5. The results presented in thisreport, especially when compared to the corresponding results in the original report, should This report follows the organization of the CRASE+ technical report. The next section contains abrief overview of CRASE and CRASE5. Section III discusses the data and processes used totrain and validate the engine, while Section IV provides validation results. Section V contains a II. Background: Automated Scoring and CRASE5 Automated scoring (or automated essay scoring) is the use of a computer algorithm to emulatehand scoring behavior on constructed-response or essay items. The scoring algorithm is calledthe engine. Preparing the scoring algorithm for operational use is called training the engine.There are four parts to a scoring engine: a means of reading text data, a preprocessor thatstandardizes and initially processes the text, a means of extracting the quantitative CRASE, short for Constructed Response Automated Scoring Engine, was created in 2007 for aU.S. state’s summative assessment program. The system has since been enhanced to includescoring methods for additional types of free-response items and to incorporate newtechnologies in text processing and modeling. CRASE has been used operationally in multiple When CRASE was first used on ACT writing, it was called CRASE+and represented the fourthmajor version of the CRASE software. With the construction of a new engine that gives theengine trainer greater flexibility in developing scoring models, ACT proposed the use of This report assumes a basic familiarity with automated scoring concepts. For readers new toautomated scoring, the CRASE research team recommends the following resources: Lottridge, S., Burkhardt, A., & Boyer, M. (2020). Digital module 18: Automated scoring.Educational Measurement: Issues and Practice,39(3), 141–142.https://doi.org/10.1111/emip.12388 McCaffrey, D., Casablanca, J., Ricker-Pedley, K., Lawless, R., & Wendler, C. (2021).Bestpractices for constructed-response scoring. ETS.https://www.ets.org/content/dam/ets- Shermis, M. D., & Burstein, J. (Eds.). (2013).Handbook of automated essay evaluation: Currentapplications and new directions.Routledge. Wood, S., Yao, E., Haisfield, L., & Lottridge, S. (2021).Establishing standards of best practice inautomated scoring. ACT. Yan, D., Rupp, A. A., & Foltz, P. W. (Eds.). (2020).Handbook of automated scoring: Theory intopractice. CRC Press. III. Methods for Engine Training and Validation Data Data from hand-scored essays are required to train the CRASE engine. These data should becollected under authentic testing conditions, if possible, and must be representative of the This report uses the same ACT writing data from the original training and validation studiesconducted prior to 2023. Readers can find details about the data, the training sample, and theblind-validation sample in theCRASE+®for ACT Writing Technical Report. This section will The training and blind-validation essays came from three sources: the September 2020 ACTInternational administration, the October 2020 ACT International administration, and selectedSpring 2021 State and District administrations. Approximately two thirds of the records camefrom the State and District administrations. Only essays obtained via online administrations Information about hand scoring score point distributions, examinee gender, examinee Hispanicstatus, and examinee race/ethnicity can be found in the CRASE+ technical report on pages 5 Training and Validation Samples Recall that a generic scoring model is an automated scoring model built using essay data frommultiple writing prompts with the goal of using the model on essay data from comparable writingprompts. (The alternative is a prompt-specific model, where the model is built using essay datafrom a single writing prompt with the goal of using the model on essay data from only thatprompt.) Generic scoring models allow for consistent scoring regardl