Mark Schneider
Commissioner, National Center for Education Statistics

Benefits and Limitations of States Benchmarking to International Standards:
A Meeting to Assist States in Making Informed Decisions about Participating in International Assessments
May 30, 2008

INTRODUCTORY REMARKS

Over the past two decades, large-scale international assessments of student performance have become well-established programs in which the United States and many other countries around the world participate. These international assessments allow us to "benchmark" the knowledge and skills of our students against students in other countries in key subjects such as mathematics and science. The United States conducts these assessments at the national level; only occasionally have these tests been conducted at the state level.

However, we know from the National Assessment of Educational Progress (NAEP) that national averages of student performance can mask significant variation across the states, and this is almost certainly true for U.S. performance on international assessments as well. In the United States, many individual states have expressed interest in participating in international assessments in order to find out how their students compare to their international peers. Like national policy makers, state leaders are driven by concerns about economic competition for global markets and the skills of their current and future workforce. These are valid interests and concerns. At the same time, there are both costs and limits to what states can learn through their participation in international assessments.

In recognition of states' strong interest in international assessments and in response to their need for objective information about the range of options available for benchmarking, on May 30, 2008, NCES convened a meeting (meeting summary and materials) of key stakeholders (meeting participants) to provide information about the possibilities and limitations of state-level participation in international assessments and to begin a discussion on the key policy and political questions that states should be considering as they consider whether and how to move forward with state-level participation in these assessments.

State Participation in International Assessments

The United States participates in two international assessments in which states can participate:

• The Trends in International Mathematics and Science Study (TIMSS), which collects data on the mathematics and science achievement of 4^th- and 8^th-grade students every four years, and
• The Program for International Student Assessment (PISA), which collects data on the reading, mathematics, and science literacy of 15-year-old students every three years.

The next international benchmarking opportunity in TIMSS will be in 2011 and in PISA in 2009 (with a commitment deadline of 1 August 2008) and again in 2012. As states consider the opportunity to participate, they should keep in mind that large-scale assessments, including international programs like TIMSS and PISA, and national programs like NAEP, perform a limited set of functions exceedingly well. Most importantly, they provide a comparable measure of student performance that allows a nation or a state to assess where it stands in relation to the other participants. However, speakers at the May 30 forum raised a number of issues that need to be carefully considered before a state proceeds with an expensive testing program.

Issues for States to Consider

• The periodicity of the tests limits their ability to measure national or state progress in time frames that are meaningful for policy makers. While NAEP measures critical subjects such as mathematics every other year, including subscores that provide states valuable diagnostic information, TIMSS is every 4 years. And while PISA is on a 3-year cycle, it measures only one subject in depth at a time. Thus PISA subscores in mathematics, science and reading are provided only once every 9 years. This means, for example, that if a state participates in PISA and institutes changes in education policy or practice to respond to specific deficiencies in mathematics skills that have been identified among its 15-year-olds based on PISA, it will be nearly a decade before the state will be able to determine if its 15-year-olds are performing better on those subskills.
• International assessments are not useful for comparisons of the performance of racial/ethnic groups and language minorities and students with disabilities because there is no agreed upon way to make these categorizations that applies across all countries. This means that international results for key reporting groups under Federal and state policy are not available from PISA or TIMSS.
• International assessments are cross-sectional, they do not measure individual student progress over time. Further, while NAEP and TIMSS allow policymakers to examine performance at different grades, e.g., 4^th, 8^th, and 12^th, to determine whether students are acquiring the skills they need in earlier grades to be prepared for later grades, PISA tests only 15-year-olds. In the case of PISA this means that the yield for the entire education system from preschool to age 15 is being evaluated, which makes it impossible to use PISA to identify which portions of the education pipeline are working well and which may need the most improvement.
• International assessments do not provide usable scores at the student, classroom, school, and, in most cases, district levels. This means that international assessment results cannot be used as part of an accountability system.
• Cultural and policy settings differ so extensively across countries that international assessments have limited utility for drawing conclusions about influences on student achievement. For example, what "school autonomy" means in the national education systems of most countries may be quite different from what it means in the United States where education is delivered through more than 13,000 school districts (an education organization that does not exist in most of the rest of the world). Further, statistical analyses that seek to identify which policies are associated with higher national performance on international assessments are, in the end, correlations between answers to questionnaires filled out by government officials and test scores. Correlation is not causation even under the best of analytic circumstances.
• The frameworks for international assessments (what the test is designed to test) can be quite different from each other, from NAEP, and from state assessments. Experience with standards and accountability reform in the United States suggests that alignment between standards for learning, teaching, and assessment is a key to student gains. It is important for a state to examine the framework for any international assessment in which it is considering participation and to ask whether it covers material the state is expecting its students to learn and its teachers to teach. If not, and if the state is not prepared to shift its standards to align with the international assessment framework, the value of the international assessment score for the state might be questioned.
• Participation in international assessments is expensive. Estimates for participation in PISA are around $0.5 million per state, and participation will increase the testing burden on students and schools.

Considering Benchmarking Alternatives

The limits and costs of states' participation in international benchmarking do not, however, erase the need and desire for international benchmarking information. NCES has been exploring alternate ways to generate state TIMSS and PISA scores that would be less expensive and less burdensome than individual state participation. These methodologies rely on their own sets of assumptions and limitations, though, which state policy makers will also have to consider and be comfortable with.

One of the most promising methods for obtaining international benchmarking information is small-area estimation, which would provide estimates of state performance from the national sample, rather than individual state samples, as above. Some analysis has been done using the current national PISA sample, and state-level PISA scores have been estimated, albeit with large confidence intervals around the estimates. NCES is working on ways to improve the precision of these estimates.

Another method is statistical linking studies, which involve expressing the scores of one assessment (e.g., NAEP) in the metric of another (e.g., TIMSS). Indeed, NAEP scores have been successfully converted to TIMSS scores, to allow states to see where they fall in the TIMSS country rankings. This approach is inexpensive and does not require the administration of the international assessment to large numbers of students in individual states. While TIMSS and NAEP are suitable for linking studies, linking PISA and NAEP presents larger problems.

Both of these methods produce state level estimates that are based on models and so the audience for the results will have to be comfortable with the assumptions of the technique.

These points and others were discussed during the meeting. Attached you will find a more detailed summary of the meeting and copies of the presentations that were delivered.

State policy makers are encouraged to access the materials from this meeting, consider the questions posed in this document, and contact NCES with any additional questions or comments.

Top