Ph.D. Student Jieun Kim recently published “Memorizing and fabricating? Uncovering high-stakes writing test preparation” in refereed journal Language Education and Assessment.
Abstract:
With high-stakes decisions relying on standardized language tests (Chapelle, 2021a), many test-takers seek test preparation from commercial “cram schools,” including TOEFL iBT writing (E.-Y. J. Kim, 2017; S. Kim, 2021). While previous studies have relied on interviews and surveys to investigate Korean students’ test preparation practices, the present study directly observed ten prerecorded online lectures from two popular cram schools with a combined revenue exceeding $150 million in 2019. The analyzed lectures were publicly available, representing their typical instructional practices. The instructional content of the lectures was coded based on Messick’s (1982) taxonomy of test preparation, revealing three types of instructional activities: Type 1 (enhancing construct-relevant factors), Type 2 (reducing construct-irrelevant factors), and Type 3 (enhancing construct-irrelevant factors) test preparation activities. Type 1 activities primarily covered vocabulary, grammar, and organizational patterns. Type 2 focused on teaching test formats, TOEFL trends, time management, and topic choice. Type 3 addressed memorizing templates, fabricating examples, gaming the scoring system, and providing cramming tips. The lectures seemed to pay enough attention to enhancing construct-irrelevant factors as much as enhancing construct-relevant factors. To increase the test scores, students were encouraged to utilize memorized templates and fabricate examples. However, these practices may undermine support for the explanation and consequence implication inferences of the validity argument of TOEFL iBT. Particularly, this test preparation method raises concerns about whether scores, relying on memorized templates and fabricated evidence, truly reflect academic writing ability. This study provides valuable insights into language testing preparation, applicable to various contexts with different tests.
This publication was based on one of her Qualifying Papers for the PhD program.
Awesome work, Jieun!