The Examination of Realiability of Scoring Rubrics Regarding Raters with Different Experience Years

Abstract

The main goal of this study is to determine the role of experience years in using scoring rubric on inter-rater reliability. In regards to this objective; the results of G and Phi studies that were obtained by three groups of raters in one of which there are raters who have much experience in using scoring rubric; in one of which there are raters who have little experience in using scoring rubric and in one of which there is a rater with much experience and one with little experience in evaluating students using scoring rubric in the process of written performance assessment with (s:r)xi design were compared. Raters in three different groups evaluated an open ended achievement test developed to test students’ written English written skills with the same scoring rubric. 120 students who study at the prep-class of a private university and 12 instructors that work at the same school participated to the study. In the study, a nested ((b:p)xm) design in which individuals are nested in raters but individuals and raters are crossed with items and raters was utilized. Based on the findings, it is found out that experience year in using scoring rubric doesn’t affect reliability when a qualified scoring rubric is employed.