Additional Collected InformationFor the 2000 state assessment, additional information was collected by quality control monitors (QCMs) about assessment administrators' (AA) handling of the specific assessment tasks. QCMs and other NAEP staff collected a variety of other useful information about the assessment procedures. This included AAs' job titles and reactions to the assessment, observations made by the QCMs, and reactions of the state coordinators. Assessment Administrator ReactionsQCMs asked AAs of both monitored and unmonitored sessions a set of questions to obtain their overall impressions of the assessment and suggestions for improving it. For monitored sessions, this was done as soon as the materials were packed for shipping. For unmonitored schools, the debriefing questions were covered during the post-assessment telephone call. AAs were asked to comment on the assessment sessions, the training sessions, the NAEP materials and forms, and the AA manuals. About Assessment AdministratorsQCMs asked each AAs for his or her job title at the school. More than 50 percent of the AAs in grade 8 public schools were counselors, compared to less than 29 percent of the AAs in grade 4. Most AAs were school-based (rather than district- or state-based), although about 14 percent of schools at grade 4, and 11 percent at grade 8, used district- or state-level staff for NAEP. Instructions to state coordinators specified that teachers who taught the subject of the assessment to the sampled students should not function as AAs to avoid any potential conflict of interest. This instruction was widely observed, as most AAs reported that they did not teach either mathematics or science to the assessed group.
Summary Observations Made by the Quality Control MonitorsQCMs made various general observations and summary evaluations about the monitored schools upon leaving the school. In the monitored schools, QCMs observed whether the assessment environment was adequate or inadequate, based on factors such as room size, seating arrangements, noise from hallways or adjacent rooms, and lighting. Of the 4,177 monitored sessions where the QCMs recorded observations, QCMs felt that 96 percent of the sessions were held in suitable surroundings. QCMs were also asked to rate the attitude of AAs, other school staff, and students toward the assessment. QCMs felt most positive about the attitudes of the AAs but received a more mixed impression about the attitudes of other school staff and the students. QCMs reported more positive attitudes from grade 4 students than grade 8 students. QCMs' were then asked to provide an overall rating of AAs' self-reliance. For monitored sessions, 87 to 89 percent of the AAs in monitored sessions were self-reliant or needed to consult the QCMs for only one or two minor items. Only about 2 percent had serious difficulty conducting the session. Comments and Reactions of State CoordinatorsUpon conclusion of the assessments, NAEP data collection staff mailed state coordinators a short survey form to obtain their reactions to the operations associated with the 2000 state assessment and any suggestions they had for improving the program. Twenty-nine state coordinators (62 percent) responded by returning a questionnaire or providing responses over the telephone. A detailed summary of the state coordinators' responses is contained in the Report on Data Collection Activities for All States (Westat 2000), which was distributed to state coordinators in October 2000. Some of the findings are reported below:
Last updated 17 June 2008 (MH) |
||||||||||||
