The following response was received from Chris Schatschneider and is posted on his behalf. 

 

Yes, there are two things that need to be addressed.  One is the "interval 
or ratio scale" issue and the other is the rate of gain issue.  Let's talk 
about the rate of gain issue first. 

The claim was made that a one grade level gain was made for every 10-13 
hours of instruction.  This claim is based upon looking at students who had 
1 hour to 78 hours of instruction (with a mean of 27 hours of instruction). 
Additionally, over 80% of the students received more that 10 hours of 
instruction.  For these students, the rate of gain is an interpolation, not 
an extrapolation.  To understand the difference, let's use the example of 
gas mileage on a car.  Let's say I fill up my car with 15 gallons of gas and 
drive it 300 miles until it runs out.  I could figure out the cars Miles Per 
Gallon (MPG) by dividing 300 by 15 - and get 20 MPG. Does that mean that 
every 20 miles I used up a gallon of gas?  Probably not.  Some miles are 
highway miles, some in the city, some up hill, some down hill.  But on 
average, I can expect to get about 20 miles to the gallon.  It's a good 
interpolation.  It's the same with grade level gains by hours of 
instruction.  Does it mean that for every hour of instruction you will gain 
.08 grade level units?  Probably not.  Some hours may produce smaller gains, 
while others produce larger gains.  But on average, it's a good 
interpolation. 

The preceding example was a case of interpolation.  It's an interpolation 
because the car being driven went farther than the 20 miles necessary to 
burn a gallon of gas.  Had we driven the car only 10 miles and noticed that 
it burned only half a tank of gas, then try to estimate MPG, that would have 
been an extrapolation. Extrapolations are more difficult to assess because 
what you are trying to estimate is outside of the range of the data.  These 
are riskier because they are more prone to individual fluctuations that 
haven't been "averaged out".  So in the present report, about 20% of the 
sample contributed to the estimate by extrapolation.  

So perhaps an important question might be "of those that got at least 10 
hours of reading instruction, how many grade level units did they gain? 
Looking that the data from NCFL, it appears that on average, they gained .07 
grade levels per hour, or about 1 grade level for 14 hours of instruction - 
which is very similar to the original estimate.  So the 20% of the sample 
that contributed to the estimate by extrapolation did not significantly 
change the result. 

The second issue brought up was the notion that the scale of grade level is 
not an interval or ratio scale, so one is not allowed to use arithmetic 
operations such as means.  The idea that grade level is not on an interval 
scale is most likely true.  But along the same lines, it is also true for 
such measures as IQ, reading achievement, and any other cognitive test that 
did not use Item Response Theory in its development.  Keeping to scales that 
only have interval and ratio properties in a study would rule out the vast 
majority of studies performed in the social sciences.  But an important 
question is how closely do grade equivalents approximate an interval scale. 
If you believe that it does, then the use of them in an analysis that uses 
means is probably warranted. 



Chris Schatschneider, Ph.D. 
Associate Professor, Department of Psychology 
Associate Director, Florida Center for Reading Research 
Department of Psychology 
1107 W. Call Street 
Office A417 
Tallahassee, Fl 32303-4301 
(w) 850-644-4323 
email: schatschneider at psy.fsu.edu 


________________________________

From: familyliteracy-bounces at nifl.gov on behalf of tsticht at znet.com
Sent: Wed 6/18/2008 3:02 PM
To: familyliteracy at nifl.gov
Subject: [FamilyLiteracy 1126] Re: Family literacy findings released todayshow one grade level gain for every 10-13 hours of instruction



Betsy and Sandy: You are not the only ones confused by the reporting of this
research. In reading the Press Release on the home page of the National
Center for Family Literacy (NCFL)  I was surprised to read: Quote" A
National Center for Family Literacy study found that adult literacy results
can improve as much as one grade level gain for an average of every 10 hours
to 13 hours of instruction." The Press Release also stated that there were
278 adults participating, so it seemed like a good size research project.

I suspect that I was not alone in thinking that in this study 278 adult
literacy learners were post-tested after 10 to 13 hours of reading
instruction and they made 1 year of gain. Then they were post-tested again
after another 10-13 hours of instruction and made another year of gain,
bringing their improvement to 2 years of gain, and so forth. I wondered if
this was what had happened.

Fortunately I learned on the NIFL Family Literacy list that there was a
brief technical report about the study on the www.famlit.org web site so I
read it. This gave me some other surprises which were not along the lines
that Susan McShane mentioned but along lines that render questionable the
research findings as reported.

First, the data on reading gains were not based on 278 adults but just 82
(39 Year 1, 43 Year 2), a 70 percent drop in the number of adults I thought
were contributing pre- and post-test data.

Second, I found that instead of having tested students after  10-13 hours,
the report said: Quote:" Hence, it seemed reasonable to administer the TABE
after 20 hours of  reading instruction and that is what the teachers did. "
End Quote So I wondered how the conclusion was reported that they made a 1
grade level gain for every 10-13 hours of instruction if teachers tested
after 20 hours.

Third, I posted a question about the pre-post-testing on the NIFL Family
Literacy list and Laura Westberg, Director, Research/Special Projects at
NCFL said that the report saying that students were post-tested after 20
hours wasn't true, and instead reading instructional hours varied from 1 to
78 across the 2 years of the study!

Fourth, Laura said that: Quote:",,, to better understand the types of gains
that students were making in relation to the amount of reading instruction
they received, we calculated a gain per instructional hour based on the
total reading instructional hours reported by the teachers for the
students. Year 1 students had an average gain per instructional hour of .10
and Year 2 students had an average gain of .08. Thus, in year 1 for every
one hour of reading instruction, they gained an average of .10 grade levels
and in year 2 for every one hour of reading instruction they gained an
average of .08 grade levels. This translates into an average of 10-12.5
hours of reading instruction to raise a student's score on the TABE by one
grade level." End Quote

So actually adults didn't make a year's gain for every 10-13 hours, as the
Press Release stated. Instead, this is a hypothetical gain that would be
made if learning progressed equally hour by hour at a rate of .10 or .08
grade levels per hour (i.e., if you multiply .10 by 10 you get 1.0 year of
gain per 10 hours of instruction, etc.).  Of course, this is an altogether
unlikely event, especially given the use of a grade level scale which is
not a ratio or interval scale, meaning doing these sorts of dividing and
multiplying of the numbers are not valid arithmetical procedures.

David Rosen asked: Quote: "I wonder if anyone can shed some light on these
NCFL findings. Other studies I have seen suggest that anywhere from 50-150
hours are needed for each grade level gain. These findings are
extraordinary." End quote We now see that the findings are questionable
because of the problems mentioned above.


Tom Mechem thought the "extraordinary" findings of the NCFL study might
reflect the use of the diagnostic/prescriptive nature of the teaching of
specific  reading components of the NCFL instructional method. But, as I
have mentioned in earlier postings, other programs not following this
method have reported gains in the range presented by the NCFL study.

For examples of other studies reporting various amounts of gain per hours of
instruction see Part 3 of  Adult Literacy in the United States: A compendium
of quantitative data and interpretive comments by Sticht & Armstrong. You
can download a copy of this report at the www.nald.ca web site in the
library pages. You can also get to it by googling it.

At the present time, based on what has been reported so far, I believe adult
literacy educators should be wary and cautious in using this NCFL reading
research in promoting adult literacy education or making changes in
education programs.


Tom Sticht



----------------------------------------------------
National Institute for Literacy
Family Literacy mailing list
FamilyLiteracy at nifl.gov
To unsubscribe or change your subscription settings, please go to http://www.nifl.gov/mailman/listinfo/familyliteracy
Email delivered to gprice at famlit.org


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/ms-tnef
Size: 10808 bytes
Desc: not available
Url : http://www.nifl.gov/pipermail/familyliteracy/attachments/20080619/b6f27511/attachment.bin