PROPOSITION 227 AND SKYROCKETING TEST SCORES: AN URBAN LEGEND FROM CALIFORNIA

Educational Leadership December 2004/January 2005
Vol 62(4): 37-39

Stephen Krashen
University of Southern California

"An Urban Legend is usually a good/captivating titillating / engrossing / incredible / worrying) story that has had a wide audience, is circulated spontaneously, has been told in several forms, and which many have chosen to believe (whether actively or passively) despite the lack of actual evidence to substantiate the story." (Urban Legends Research Centre, www. urla.com)

I wish to add another Urban Legend to those that already exist, a legend that in my opinion ranks with the legend of alligators in the sewers of New York City.¹ It is the "Skyrocket Legend": As a result of dropping bilingual education, test scores in California "skyrocketed."

This legend has had serious consequences. The Skyrocket Legend was interpreted by many as a demonstration of the superiority of immersion over bilingual education, and encouraged anti-bilingual education advocates to eliminate bilingual education in other states. I will argue here that dropping bilingual education did not contribute to the increase in test scores, that there are more reasonable explanations for the test score increase and I will review evidence showing that bilingual education is a very helpful idea.

Bogus means of increasing test scores

Why did test scores go up in California? Proposition 227 took force in 1998, at the same time the new SAT9 test was introduced. Research (Linn, Graue, and Sanders, 1990) has shown that after new tests are introduced, test scores rise, which is why commercial tests need to be recalibrated every few years. "Test inflation" was especially prevalent in California where the same test had been given for several years in a row, punishments for lower scores were severe, and rewards for higher scores were generous, at least while there was money in the budget. This pressure resulted in districts using unusual and extraordinary means for raising test scores, some of which have nothing to do with increased competence.

Among the bogus means of increasing test scores are extensive training in certain test-taking skills and selective testing, i.e. excluding low scoring children from taking the test. Asimov (2000) suggested that selective testing may have occurred in California. She reported that in many cases in which SAT9 scores increased from year to year, the number of students tested decreased. According to Asimov, "questionable pairings" appeared in 22 San Francisco Area school districts. And of course the use of some test-taking strategies will raise scores without an increase in competence: If there is no penalty for guessing, for example, simply encouraging guessing will raise scores. Use of these means to raise scores is like claiming to raise the temperature of the room by lighting a match under the thermometer.

Test scores and 227

Parrish, Linquanti, Merickel, Quick, Laird and Esra (2002) compared test scores for California English learners in 682 schools that kept bilingual education with 1184 that did not. Their analysis included students currently considered English language learners as well as those who were formerly English language learners but who were subsequently reclassified as proficient.

Parrish et. al. provided two analyses: In a "quasi-longitudinal" study they analyzed SAT9 results for English learners who were in grade 2 in 1998 and in grade 5 in 2001. This is considered only "quasi-longitudinal" because the two groups were not identical, because of transiency. There was practically no difference in gains between those in schools that kept bilingual education and those in schools that dropped bilingual education. In schools that kept bilingual education, the gain was from 537 to 621, 84 points. In those that dropped bilingual education, the gain was from 540 to 623, 83 points.

A second analysis was cross-sectional. In 1998, fifth graders in bilingual education scored 613, and fifth graders in schools that were about to drop bilingual education scored 614. In 2001, the fifth grade scores were 621 for schools that kept bilingual education and 623 for schools that did not.

Not all English immersion was pure English immersion, however. Parrish et. al. reported that for 75 districts that said they used "structured English immersion," 68% reported that teachers used the child's first language "occasionally or frequently" for academic content instruction, 88% used it occasionally or frequently for "preview-review," that is, presenting information first in the primary language as a preview, and 48% reported that they used instructional aides who spoke the first language occasionally or frequently. Their comparison, thus, appears to have been between bilingual programs and programs that used aspects of bilingual education.

The Bali study

Rossell (2002) rejects the Parrish et. al. results because more children were tested in 2001 than in 1998. She cites with approval, however, Bali (2001), who provides data from one district, Pasadena, and controlled for different rates of testing of children. Bali compared the progress of children who had been in bilingual education but who, because of Proposition 227, were abruptly placed in all-English classes, with those who had been in all-English classes all along. Bali reported that students who had been in bilingual education gained 4.15 points in reading the year after leaving bilingual education, compared to gains of 1.8 points for those who had been in English previously, with no difference in math. Bali and Rossell interpret this finding as showing that Prop. 227 did no harm and in fact did some good. But one could easily argue that the superior gains of those formerly in bilingual education were due to bilingual education: In bilingual education, students learned subject matter knowledge through their first language, which made the English they subsequently heard more comprehensible, and developed literacy in their first language, which facilitated subsequent literacy development in the second language.²

There is thus no evidence that dropping bilingual education resulted in significantly more "English for the children." California, it seems, dismantled a program that allowed for equivalent English language development with less time actually devoted to English, a stunning confirmation of the efficacy of providing first language instruction. California gave up the advantages of bilingualism and received nothing in return.

Why bilingual education helps English language development

Briefly, quality bilingual programs introduce English right away and teach subject matter in English as soon as it can be made comprehensible, but they also develop literacy in the first language and teach subject matter in the first language in early stages.

As noted earlier, developing literacy in the first language is a short cut to English literacy. It is much easier to learn to read in a language one understands, and once a child can read in the primary language, reading ability transfers rapidly to English.

Teaching subject matter in the first language stimulates intellectual development and provides students with valuable knowledge that will help the child understand instruction when it is presented in English.

What controlled studies say

A valid way to determine the effect of bilingual education is to perform controlled studies. In these studies, programs are compared in which the only difference is the use of the first language.

Such scientifically valid controlled studies have been done, and they consistently show that students in properly organized bilingual programs acquire at least as much English as comparison students in all-English programs, and usually acquire more. Willig (1985), Greene (1997) and Slavin and Cheung (2005) conducted reviews of the research in bilingual education using statistical tools far more precise and sensitive than those used in other reviews, and concluded that the use of the first language in instructing limited English proficient children has beneficial effects on English language development.

A recent study of the effectiveness of bilingual education was done by a research team headed by K. Oller and Eilers (2002). At grade five, students in a bilingual program (60% English, 40% Spanish) did as well as comparisons in an all-English program (with an optional 10% of the day in Spanish) on tests of English literacy, and did far better on tests of Spanish.

Studies from other countries give results that are very consistent with those from the United States. Children in well-organized bilingual programs acquire as much of the second language as those in "immersion" programs or more. Studies confirming this have been done with Turkish and Urdu speaking children in Norway, Punjabi speaking children in England, Turkish and Arabic speaking children in the Netherlands, Finnish-speaking children in Sweden, Gapapuyngu speaking children in Australia, and Tzeltal and Tzotzil speaking children in Mexico (Krashen, 1999a).

Rossell and Baker (1996) have also reviewed the research on bilingual education but have concluded that bilingual programs are not as effective as all-English immersion programs. Their review, however, inappropriately excluded a number of valid studies, and inappropriately included studies that were not valid comparisons, such as comparisons of different types of Canadian Immersion programs. Some of the programs they labeled "immersion" were, in fact, bilingual programs. (Krashen, 1966,1999b).

Even so, Rossell and Baker conclude that "additional, methodologically sound research needs to be conducted in order for the courts and policymakers to make intelligent decisions" (p. 39) and that "we are struck by how small the differences are ... between programs with very different amounts of English instruction" (p. 43). The Rossell and Baker review is by far the most negative review of bilingual education published; in fact, it is the only one I know of that claims that all-English alternatives are better. It concludes, however, that differences are not huge and that more research is necessary in order to make "intelligent decisions."

Clearly, the published research is not consistent with claims that dropping bilingual education causes scores to "skyrocket" and does not support movements to dismantle bilingual education.

Summary

There is no question that test scores went up in California, but dropping bilingual education had nothing to do with the increase. Test score increases in California appear to be a result of the usual "test score inflation" that occurs when new tests are introduced. In California, inflation has been particularly strong because of intense pressure to raise scores.

Test scores for everyone went up in California. There were no clear differences in gains between districts that kept bilingual education and those that dropped or reduced their bilingual programs.

Missing from nearly all discussions of the effectiveness of bilingual education is the consistent finding that controlled studies show that bilingual education works. The Skyrocket Urban Legend is false.

Notes

Other Urban Legends include: Humphrey Bogart was the original Gerber baby on their baby food ads, and if the entire population of China jumped up at the same time, the US would be swamped by a tidal wave. They are not true. See www.urbanlegends.com for many others.

Bali may have been influenced by the fact that in 1998 those in bilingual education in Pasadena had slightly lower scores than English learners in other programs. This was probably because children with less English proficiency when entering school tend to be placed in bilingual rather than all-English classes. The only data in Bali's study that is properly statistically controlled are the gains made after all English learners were put in all-English classes. Those with previous bilingual education made better gains. This suggests that bilingual education prepared them better for all-English than all-English. We must, however, be cautious. The gains were small, and Bali presents no description of the kind of bilingual program Pasadena had, nor does she tell us very much about the all-English classrooms. What is clear is that her data does not support her conclusion that stopping bilingual education "actually helped." Rather, the data suggest that doing bilingual education helped.

References

Asimov, N. (2000, Saturday, July 22). Test Scores Up, Test-Takers Down: Link between participation, improvement on school exam prompts concern. San Francisco Chronicle.

Bali, V. (2001). Sink or swim: What happened to California's bilingual students after Proposition 227? State Politics and Policy Quarterly, Spring, 2001: 295-317.

Greene, J. (1997). A Meta-analysis of the Rossell and Baker review of bilingual education research. Bilingual Research Journal, 21(2,3): 103-122.

Krashen, S. (1996). Under attack: The case against bilingual education. Culver City, CA: Language Education Associates.

Krashen, S. (1999a). Condemned without a trial: Bogus arguments against bilingual education. Portsmouth, NH: Heinemann.

Krashen, S. (1999b). Why Malherbe (1946) is NOT evidence against bilingual education. NABE News, 22(7): 25-26.

Linn, R., Graue, E., & Sanders, N. (1990). Comparing state and district test results to national norms: The validity of claims that "everyone is above average." Educational Measurement: Issues and Practice, 10, 5-14.

Oller, K. & Eilers, R. (Eds.). (2002). Language and literacy in bilingual children. Clevedon: Multilingual Matters.

Parrish, T., Linquanti, R., Merickel, A., Quick, H., Laird, J. & Esra, P. (2002). Effects of the implementation of Proposition 227 on the education of English learners, K-12: Year 2 report. San Francisco: West Ed.

Rossell, C. (2002). Dismantling bilingual education, implementing English immersion: The California initiative. San Francisco: Public Policy Institute.

Rossell, C., & Baker, K. (1996). Bilingual Education in Massachusetts: The Emperor has no clothes. Boston: The Pioneer Institute for Public Policy Research.

Slavin, R. and Cheung, A. 2005. A synthesis of research of reading instruction for English language learners, Review of Educational Research, 75(2): 247-284.

Willig, A. (1985). A meta-analysis of selected studies on the effectiveness of bilingual education. Review of Educational Research, 55: 269-316.