Training for the Scoring Raters
Training of NAEP scoring raters is conducted by NAEP item development and scoring staff. All assessments are scored item-by-item so that each scorer works from only one scoring guide at a time. After scoring all available responses for an item, a team then proceeds with training and scoring of the next item. Training for current assessment scoring involves explaining the item and its scoring guide to the team and discussing responses that represent the various score points in the guide. The trainer provides three or four student responses to "anchor," each score point. When review of the anchor responses is completed, the raters score 10 to 20 pre-scored "practice papers" that represent the entire range of score points the item could receive. The trainer then leads the team in a discussion of the practice papers to focus the raters on the interpretation of the scoring guide. After the trainer and supervisor determine that the team has reached consensus, the supervisor releases work on the image-based scoring system for the raters. The raters initially gather around a PC terminal to group-score items to ensure further consensus. Following group-scoring, raters work in pairs as a final check before beginning work individually. Once the practice session is completed, the formal scoring process begins. During training, raters and the supervisor keep notes of scoring decisions made by the trainer. The scoring supervisor is then responsible for compiling these notes and ensuring that all raters are in alignment; this process in referred to as calibration in NAEP. Teams vary greatly in the amount of time spent scoring as a group before working individually. Training for trend scoring is only slightly different in that prior year trend papers must be reviewed to understand scoring decisions made in prior years before raters can commence further scoring. Last updated 12 June 2008 (TS) |
||||||||
