 |
|
NCES: 2002004 August 2002 |
PDF & Related InfoThe Fast Response Survey System (FRSS) was established in 1975 by the National Center for Education Statistics (NCES), U.S. Department of Education. The FRSS is designed to collect small amounts of issue-oriented data with minimal burden on respondents and within a relatively short timeframe. Surveys are generally limited to three pages of questions, with a response burden of about 30 minutes per respondent. Sample sizes are relatively small (usually about 1,000 to 1,500 respondents per survey) so that data collection can be completed quickly. Data are weighted to produce national estimates of the sampled education sector. The sample size permits limited breakouts by classification variables. However, as the number of categories within the classification variables increases, the sample size within categories decreases, which results in larger sampling errors for the breakouts by classification variables. The FRSS collects data from state education agencies, local education agencies, public and private elementary and secondary schools, public school teachers, and public libraries.
Before the main survey was mailed out, a pilot study was conducted. Given the lack of available information about the numbers of alternative programs across the nation (the Common Core of Data (CCD) only includes data on alternative schools), the pilot study aimed to determine the number of alternative programs that existed in regular districts31 both with and without alternative schools. The results of the pilot study were used to inform the main study's sample to increase the likelihood that the districts sampled would be representative of the nation's districts with alternative schools and programs for at-risk students and provide a sufficient number of cases to allow breakouts of results by classification variables (such as district size and region). In addition, it was anticipated that the pilot study would shed light on the extent to which the 1998- 99 NCES CCD was up-to-date and complete with respect to information on the nation's alternative schools. Three hundred and thirty-seven districts from the 1998-99 NCES CCD Public Universe File were selected for the pilot.
Based on the results of the pilot study, it was concluded that an estimated 45 to 55 percent of the districts in the CCD file had at least one alternative school or program. Moreover, the information available in the 1998-99 CCD file about the presence of alternative schools was not in line with the pilot study results. For example, the pilot study revealed that while 87 percent of districts did not report any alternative schools in the CCD, over 40 percent of these actually had at least one alternative school. Further, among the 11 percent of districts (about 1,800) that reported one or more alternative schools in the CCD, about 10 percent did not operate such schools at the time of the pilot study. These differences may have been due to the time elapsed between 1998-99 and 2000-01; alternative education is variable and fluid, and while many districts may have established new alternative schools between 1998 and 2001, others may have eliminated them. Also, there may have been differences in the definitions of alternative schools employed for the pilot study and for the CCD (e.g., unlike the CCD, the pilot study was limited to alternative schools for students at risk of education failure). The implication of these results was that considerable "oversampling" was required to obtain the desired number of eligible districts for analysis purposes.
Information from the pilot study helped guide the allocation of the total sample to the two major categories of districts: districts that reported alternative schools in the CCD and those that did not report alternative schools in the CCD. Within each category, the samples were further allocated to district size strata (less than 2,500, 2,500 to 9,999, 10,000 or more) in rough proportion to the aggregate square root of the enrollment in the stratum. The sampling frame was also ordered by metropolitan status (urban, suburban, rural) and region (Northeast, Southeast, Central, West) to induce additional implicit stratification. Within each primary stratum, districts were selected systematically and with equal probabilities.
The sampling frame constructed consisted of 14,619 regular public school districts during the 1998-99 school year. After the stratum sample sizes were determined, a final sample of 1,609 districts was systematically selected from the sorted file using independent random starts. The 50 states and the District of Columbia were included in the sample, while school districts in the outlying U.S. territories were excluded. Districts are of three types: unified, secondary, and elementary. Unified districts serve students across all grade levels and comprised 83 percent of the total sample (table A-1). Secondary districts comprised 2 percent, and elementary districts (i.e., serving grades no higher than grade 8) comprised 15 percent of the sample.
Questionnaires and cover letters were mailed to districts in January 2001. The cover letter indicated that the questionnaire was to be completed by the district-level personnel most knowledgeable about the district's alternative schools and programs. The cover letter also indicated that collaboration was encouraged if needed.
Telephone followup was conducted from mid- February 2001 through mid-April 2001 for districts that did not respond to the initial questionnaire mailing. Completed questionnaires were received from 1,540 districts. Of the 1,540 districts that completed surveys, 6 were later excluded from the sample after determining that they were not regular districts, but rather, were "regional" districts that served multiple districts and special populations of students (i.e., at-risk or special education). The weighted response rate was 97 percent. Weighted item nonresponse rates for 93 percent of individual questionnaire items were below 1 percent. Weighted item nonresponse rates for the remaining 7 percent (8 questionnaire items) ranged between 1 to 2.6 percent.
In addition to the survey questionnaires, respondents were asked to complete lists of the alternative schools and programs in their districts (if applicable). Data retrieval included telephone follow-up calls for lists that did not include the same number of schools and programs as reported in question 2 of the survey, as well as for lists that included schools or programs that did not appear to fit the survey definition (i.e., ABC Magnet School, or XYZ School for the Gifted and Talented). The weighted response rate for the list collection was 97 percent.
The responses were weighted to produce national estimates (see table A-2). The weights were designed to adjust for the variable probabilities of selection and differential nonresponse. The findings in this report are estimates based on the sample selected and, consequently, are subject to sampling variability.
The survey estimates are also subject to nonsampling errors that can arise because of nonobservation (nonresponse or noncoverage) errors, errors of reporting, and errors made in data collection. These errors can sometimes bias the data. Nonsampling errors may include such problems as misrecording of responses; incorrect editing and coding; differences related to the particular time the survey was conducted; or errors in data preparation. While general sampling theory can be used in part to determine how to estimate the sampling variability of a statistic, nonsampling errors are not easy to measure and, for measurement purposes, usually require that an experiment be conducted as part of the data collection procedures or that data external to the study be used.
To minimize the potential for nonsampling errors, the questions were pretested with respondents like those who completed the questionnaire. During the design of the survey and survey pretest, an effort was made to check for consistency of interpretation of questions and to eliminate ambiguous items. The questionnaire and instructions were extensively reviewed by the National Center for Education Statistics, the Office of Special Education and Rehabilitative Services (OSERS), and the Office of Elementary and Secondary Education (OESE), U.S. Department of Education. Manual and machine editing of the questionnaire responses were conducted to check the data for accuracy and consistency. Cases with missing or inconsistent items were recontacted by telephone. Data were keyed with 100 percent verification.
The standard error is a measure of the variability of estimates due to sampling. It indicates the variability of a sample estimate that would be obtained from all possible samples of a given design and size. Standard errors are used as a measure of the precision expected from a particular sample. If all possible samples were surveyed under similar conditions, intervals of 1.96 standard errors below to 1.96 standard errors above a particular statistic would include the true population parameter being estimated in about 95 percent of the samples. This is a 95 percent confidence interval. For example, the estimated percentage of suburban districts that reported having alternative schools or programs during the 2000-01 school year was 40.8 percent, and the estimated standard error was 2.09 percent. The 95 percent confidence interval for the statistic extends from [40.8 - (2.09 times 1.96)] to [40.8 + (2.09 times 1.96)], or from 36.7 to 44.9 percent. Tables of standard errors for each table and figure in the report are provided in appendix B.
Estimates of standard errors were computed using a technique known as jackknife replication. As with any replication method, jackknife replication involves constructing a number of subsamples (replicates) from the full sample and computing the statistic of interest for each replicate. The mean square error of the replicate estimates around the full sample estimate provides an estimate of the variances of the statistics. To construct the replications, 50 stratified subsamples of the full sample were created and then dropped one at a time to define 50 jackknife replicates. A computer program (WesVar4.0) was used to calculate the estimates of standard errors. WesVar4.0 is a stand-alone Windows application that computes sampling errors for a wide variety of statistics (totals, percents, ratios, log-odds ratios, general functions of estimates in tables, linear regression parameters, and logistic regression parameters).
The test statistics used in the analysis were calculated using the jackknife variances and thus appropriately reflected the complex nature of the sample design. In particular, an adjusted chisquare test using Satterthwaite's approximation to the design effect was used in the analysis of the two-way tables. Bonferroni adjustments were made to control for multiple comparisons where appropriate. For example, for an "experimentwise" comparison involving g pairwise comparisons, each difference was tested at the 0.05/g significance level to control for the fact that g differences were simultaneously tested. The Bonferroni adjustment results in a more conservative critical value being used when judging statistical significance. This means that a comparison that would have been significant with a critical value of 1.96 may not be significant with the more conservative critical value. For example, the critical value for comparisons between any two of the four categories of region is 2.64, rather than 1.96 which would be used for two categories. This means that there must be a larger difference between the estimates when there are multiple pairs of comparisons for there to be a statistically significant difference.
Given the importance of the definition of alternative schools and programs for this survey, and given that studies have shown that respondents do not always carefully read definitions, two steps were taken. First, as mentioned earlier, telephone followup was carried out during data collection in cases where lists did not include the same number of schools and programs as reported in question 2 of the survey. In addition, followup was conducted for lists that included schools or programs that did not appear to fit the survey definition (i.e., ABC Magnet School, or XYZ School for the Gifted and Talented). These subsequent conversations with respondents allowed for resolution of discrepancies and the removal of schools or programs from lists that did not fit the survey definition.
Second, a follow-up study was conducted to determine whether respondents had read the survey definition when completing the questionnaire. Respondents who returned questionnaires by mail or fax were of concern because there was no phone interviewer to ensure that the respondent understood and had read through the definition. Of the 848 districts that reported having at least one alternative school or program for students at risk of education failure, 771 completed the questionnaire by mail or fax. Of these, a random sample of every seventh district was selected, resulting in a sample size of 111 districts.
An initial call was made to districts in the sample to ascertain whether the original respondent was still at the district, to identify some other respondent if necessary, and to arrange an appointment. The survey definition was then faxed to the respondent. When respondents were recontacted, interviewers stated that the purpose of the call was to examine data collection procedures. After confirming that the respondent had read through the definition, respondents were asked how many alternative schools and programs were in their district during the 2000-01 school year that fit the definition. If the number reported was the same as in the original survey, the interviewer closed the interview. If the number was smaller or larger, the interviewer attempted, by referring to the schools and programs originally reported by respondents in the list collection, to ascertain why this was the case.
Of the 111 districts in the sample, data were collected from 100. Of these, 86 reported a number of alternative schools and programs that matched the number reported in the main survey. Fourteen cases did not match, and while 7 reported a number that was smaller, 7 reported a number that was larger than the one given at the time of the main survey. Of the 7 districts that reported a larger number than in the main survey, the reasons fell into 2 categories. First, in 5 cases, respondents admitted their oversight in neglecting to report alternative schools or programs in the original survey. Second, in 2 cases, respondents said that the definition was unclear or "did not sink in." Of the 7 districts that reported a smaller number, reasons given fell into two categories. In 4 cases, respondents mistakenly reported programs where students spent less than 50 percent of their instructional time. In 3 cases, respondents mistakenly reported schools or programs that were not administered by their district. There were no districts in the sample that reported a smaller number in the follow-up study because of having mistakenly included schools or programs not for at-risk students in the main survey. It may be concluded then that all of the schools and programs reported in the main survey (at least by sampled districts in the follow-up) were for students at risk of education failure.
Less than 2,500
2,500 to 9,999
10,000 or more
Northeast: | Connecticut, Delaware, District of Columbia, Maine, Maryland, Massachusetts, New Hampshire, New Jersey, New York, Pennsylvania, Rhode Island, and Vermont | |
Southeast: | Alabama, Arkansas, Florida, Georgia, Kentucky, Louisiana, Mississippi, North Carolina, South Carolina, Tennessee, Virginia, and West Virginia | |
Central: | Illinois, Indiana, Iowa, Kansas, Michigan, Minnesota, Missouri, Nebraska, North Dakota, Ohio, South Dakota, and Wisconsin | |
West: | Alaska, Arizona, California, Colorado, Hawaii, Idaho, Montana, Nevada, New Mexico, Oklahoma, Oregon, Texas, Utah, Washington, and Wyoming |
Percent minority enrollment in the school - The percent of students enrolled in the district whose race or ethnicity is classified as one of the following: American Indian or Alaska Native, Asian or Pacific Islander, Black (non-Hispanic), or Hispanic, based on data in the 1998-99 CCD file.
5 percent or less
6 to 20 percent
21 to 50 percent
More than 50 percent
Percent of students at or below the poverty level - This item served as the measurement of the concentration of poverty within the district. It is based on Title I data, which the U.S. Department of Education uses for estimates of school-age children in poverty to allocate federal funds under Title I of the Elementary and Secondary Education Act for education programs to aid disadvantaged children. The estimates are provided by the Bureau of the Census, and, for the purpose of this report, were broken into the following categories, based on the percentage of children ages 5-17 in families below the poverty level within districts in 1996- 97:
10 percent or less
11 to 20 percent
More than 20 percent
It is important to note that some of the district characteristics used for independent analyses are related to each other. For example, internal analysis of sampled districts" characteristics within the data set revealed that enrollment size and metropolitan status of districts are related, with urban districts typically being larger than rural districts (data not shown in tables). Similarly, poverty concentration and minority enrollment are related, with districts with a high minority enrollment also more likely to have a high concentration of poverty. In addition, a relationship may exist between district type (unified, elementary, and secondary) and particular district characteristics. Other relationships between analysis variables may exist.
Because of the relatively small sample used in this study, it is difficult to separate the independent effects of these variables. Their existence, however, should be considered in the interpretation of the data presented in this report.
The survey was performed under contract with Westat, using the Fast Response Survey System. Westat's Project Director was Elizabeth Farris, and the Survey Manager was Brian Kleiner. Bernie Greene was the NCES Project Officer. The data were requested by the Office of Special Education and Rehabilitative Services (OSERS) and the Office of Elementary and Secondary Education (OESE), U.S. Department of Education.
This report was reviewed by the following individuals:
For more information about the Fast Response Survey System or the district survey of alternative schools and programs, contact Bernie Greene, Early Childhood, International, and Crosscutting Studies Division, National Center for Education Statistics, Office of Educational Research and Improvement, U.S. Department of Education, 1990 K Street, NW, Washington, DC 20006, email: Bernard.Greene@ed.gov, telephone (202) 502-7348.
