Establishing Categories in the Design of Rating Scales for MA-in-ELT Theses

Abstract

This paper focuses on a data-driven method for developing rating scales. The authors describe a project in which a set of analytical scales were developed for the evaluation of MA-in-English Language-Teaching (ELT) theses. An important stage in designing assessment systems is how rating scale categories may be identified, especially in the case of a complex construct, as is determined by requirements of MA theses, and when the categories are not handed down from time-honoured traditions. In this project, there was a conscious effort to make scale development as broadly-based as possible by involving as many of the potential users of the scales as possible. In this article, the authors present insights into some of the key issues that arose during scale development. Although the rating scales are context specific, the steps illustrated here may be useful for those planning to embark on a similar project in their own institutions, as well as for experts involved in assessment in general.