New K-12 Test Data That Is Worthy of Our Attention

This report has been prepared by Scott Grimes, Director of Policy Research & Program Development, San Diego Dialogue.

Given the intense interest in the status of K-12 education reform in the San Diego region, it’s surprising that there is so little attention given to data that is reported to policy-makers and the public on student and school performance. This is especially surprising when new data comes to light that responds to the variety of critiques that are traditionally leveled at the most commonly reported indicators of student achievement.

Recently a series of new data sets were released by San Diego City Schools and presented to the school district’s Board of Trustees regarding the status of student achievement within the school system. While this data received no coverage in the press, it’s worthy of attention because it addresses many of the criticisms that outside experts offer when looking at the way that our school systems, and the state of California, measure student performance.

In this issue of San Diego Dialogue Report we look at three of the most commonly leveled criticisms of student achievement data in San Diego and describe how these new data sets help to respond to these critiques. We also look at how the Focus Schools are doing in comparison to the rest of the District, as it is these schools have received the greatest attention in the Blueprint for Student Success.

The most commonly leveled criticisms of student achievement data are:

•Criticism # 1:
State-mandated achievement exams only score students relative to each other, they don’t tell us how students are doing compared to an absolute standard.

•Criticism # 2:
Looking at student achievement data over multiple years is misleading, because the same students weren’t tested in every year.

•Criticism # 3:
Looking at the percentage of students who graduate doesn’t tell us what the graduate has achieved. We need to look at the absolute standard of performance achieved by high school students, not merely whether they earned credits and received a diploma.

Many critics (including writers in this publication) have noted that the widely reported Stanford-9 test, which is mandated for all students in the state of California, is a “norm-referenced” assessment. In other words, it compares the performance of students against a national norm, which is derived from a national sample of students who took the same exam. As a result, the Stanford-9 may tell you that a student scored in the “75th percentile” (relative to the national sample), but it doesn’t indicate whether the student knows and can do what they are supposed to, given their grade level and the academic standards adopted by the state and/or their school district.

During the past academic year the state of California, for the first time, scored and reported student performance on the California Standards Test, a set of examination questions and writing exercises administered as part of the state’s testing regimen that is benchmarked against the state’s adopted standards for English Language Arts. The results of this exam provide a new window into the status of student achievement in San Diego City Schools.

As noted in Chart A, substantial numbers of students in San Diego Unified continue to perform below the state’s adopted standards for academic achievement in this subject area. Students who scored at a “proficient” or “advanced” level are considered to be meeting the state’s standards. Students scoring at the “basic” level are considered to be close to meeting the standard, while the balance of the students score below or far below the state’s standards of achievement. As shown in the chart, about 1/3 of the District’s students are currently meeting the state’s standards in this subject area, while nearly two-thirds are scoring below standards. It’s worth noting that the District does slightly outperform the state as a whole on this assessment.

There is even greater cause for concern when looking at this assessment if the data on student achievement is disaggregated by ethnicity. This breakdown is displayed in Chart B. While approximately 1/3 of the District’s students are performing to standards, only 15% of the District’s Hispanic students, and only 20% of its African-American students, are scoring at a Proficient or Advanced level on this assessment. By contrast, 58% of the District’s White students meet the state’s performance standards in this subject area.

Criticism # 2: Looking at student achievement data over multiple years is misleading, because the same students weren’t tested in every year.

Another criticism of the Stanford-9 results, at least as they are commonly reported, is that the group of students tested changes to a certain degree from year to year. As students move through the school system they will move into, and out of, the band of students (grades 2-11) who must take the Stanford-9. Also, large urban districts like San Diego City Schools experience student “transiency,” which causes some students to be tested one year and then not tested the next because they have left the school district. Finally, in the early years of the Stanford-9 there were substantial numbers of students who were exempted from having to take the test. As these students have been brought into the testing pool they have increased and changed the composition of the set of students taking the assessment.

Due to these factors, critics of the testing regimen have called for the reporting of multi-year “matched data,” in which the performance of only those students who have taken the test every year over the last four years are included in the results. In a sense, this is a “fairer” way of looking at the school system’s outcomes, because it reports the results of only those students that the District could affect over this entire stretch of their academic careers. Recently San Diego City Schools has released a new analysis, based on matched data, which shows the gains being made by students within the system. This analysis provides additional evidence that the achievement gap, while still significant, seems to be shrinking.

Chart C displays the performance of students on the SAT-9 over the past four years for the cohort of students who were classified as “English Language Learners” in 1998. The graph shows shifts in student performance over the four years across the five performance “bands,” or quintiles, of the Stanford-9. In other words, students scoring in Q1 fell within the 1st-19th percentile ranking when compared to the national norm, while students in Q5 scored in the top 20%, based on the national sample.

As shown in the graph, between 1998 and 2001 fewer and fewer English Language Learner (ELL) students performed at the lowest levels. In 1998 67% of ELL students scored in the bottom quintile, while by 2001 this number had shrunk to 46%. The percentage of students scoring in the top two quintiles more than doubled during this time period. In the future it will be important to continue to track the performance of students using matched data, not only on the Stanford-9, but also on the California Standards Test and the District’s own standards-based assessments. These types of analysis will provide the clearest, most accurate portrait of the actual performance of school system in helping all students to achieve high standards.

Criticism # 3: Looking at the percentage of students who graduate doesn’t tell us what the graduate has achieved. We need to look at the absolute standard of performance achieved by high school students, not merely whether they earned credits and received a diploma.

Critics of the American high school are increasingly pointed in their critiques of the worth of a high school diploma. They point out that employers find many high school graduates lack the basic skills necessary to be useful as employees. Large numbers of entering college students are forced to take remedial courses, because they did not achieve the knowledge and skills in high school necessary to succeed at the collegiate level.

As a result, many states, including California, are now requiring students to pass a high school exit exam before earning a diploma.

Recently the first round of data on student performance on the new high school exit exam was released to the public. The results show the substantial efforts that still need to be undertaken to ensure that all students achieve rigorous standards of achievement before graduating from high school. In the spring of 2001 9th graders across the state were afforded the opportunity to take the new high school exit exam. In San Diego City Schools, 87% of the 9th graders in the system opted to take the test.

Chart D shows the performance of the District’s students on the assessment, as compared with the performance of students statewide. Separate results are displayed for English Language Arts and Mathematics (students most pass both subject areas to successfully pass the exam and graduate from high school.) As noted in the graph, a little over two-thirds of the students tested in San Diego passed the English component of the exam, while less than half passed the math portion.

On one hand, these results might seem encouraging. After all, these students are at the beginning of their high school careers and presumably over the next three years they will acquire additional knowledge and skills in both subject areas. But a substantial challenge remains, particularly in the area of mathematics. Also, in the larger scheme of educational reform, it’s important to note that this exam is not particularly rigorous. Passing the mathematics portion of the exam, for example, requires only mastering the standards for math set for those who have completed the eighth grade. In other words, it’s possible to pass the high school exit exam and still not master the academic standards that have been adopted by the District through the senior year of high school.

Reviewing and understanding data on student achievement in a school system, particularly one as large and diverse as San Diego City Schools, can be a laborious process. It’s often difficult to “boil down” conclusions about student progress to a few key points or findings. But looking at data like these is crucial to developing an honest set of benchmarks for a school system that is undergoing enormous change. If we are to develop a true “public” for public schools, we must develop forums and spaces where regular conversations about data can take place. We also must encourage the regional media to cover this type of information with the care and attention it deserves.