Automated essay evaluation: lessons and limitations

Douglas R McPherson
Texas A&M University
Doha, Qatar

The potential and shortcomings of computer-mediated writing (CMW) are being increasingly widely discussed. This paper aims to further this discussion, informed by an ongoing endeavour to use CMW effectively in a developmental writing programme at one institution. The types of CMW used, how they are used, reasons for exploring this field, and the challenges and limitations involved in utilizing such technology are explored.

Measurement of student learning, and the necessity for benchmarked measurement for all stakeholders, are described, particularly in the context of developmental English writing courses taught within the institution concerned. Accountability for learning, the often disparaged role of extrinsic motivation in learning, and the merits of using a cumulative assessment model in conjunction with CMW are considered.

The two web-based software services described and discussed in this paper are 'Criterion', an ETS product, and 'Writeplacer ESL', the essay evaluation software within the College Board's Accuplacer placement system. The former software is used largely formatively and the latter totally in a summative sense. Other software packages used in conjunction with this approach to writing are described, and associated pedagogical approaches are discussed.

The limitation of machines in dealing with some styles and genres of writing are briefly explored, and the paper concludes by suggesting that computer-mediated writing, despite its limitations and weaknesses, can be effective and efficient when used in conjunction with more traditional and non-technological approaches.