Elaine Garan (Garan 2002) took a close look at the panel’s report, and found that claim (1) was only true for tests in which children read lists of words in isolation. It was not true for tests of reading comprehension. In fact, for tests of reading comprehension given after grade 1, the effect of heavy phonics instruction was barely perceptible.

When we give children tests of words in isolation, they have no choice but to appeal to their knowledge of phonics; it is no wonder that intensive phonics instruction shows such a strong effect. This does not show that intensive phonics is helpful in learning to read. Smith (2003) points out that "this is like tying children’s feet together to prove they must jump before walking" (p. 13). In both cases, we have constrained the situation so that children are forced to use unnatural means of accomplishing a task.

The National Reading Panel did not distinguish between different kinds of tests used when making their second claim, the claim that skills-based instruction is superior to whole language. Some tests were measures of reading single words in isolation, some involved real texts. They also did not closely examine what went on in the treatments; The issue is not whether a treatment is labeled "whole language" or "skills" but how much reading the children actually did. In some studies, the group labeled "skills" or "traditional" actually read more than the group labeled whole language.

I re-analyzed this data (Krashen, 2002) with two alterations: (1) Considering only tests of reading comprehension. (2) Considering not whether a treatment is labeled "whole language," or "phonics" but whether the children in one treatment were actually doing more real reading than the children in the other treatment. In addition, I included some studies that the NRP had missed. My results were dramatically different from those reported by the National Reading Panel: I found an advantage favoring whole language.

My conclusion on the second National Reading Panel claim is what methodologists call "post hoc." I went back and looked at previously done studies using my own framework, asking different questions than those the original researchers had asked. Scientifically, this is not a strong way of supporting a hypothesis. What is clear, however, is that the National Reading Panel’s interpretation of the results is not the only possible one.

2