Thank you so much for clarifying the issue of reading gains.  I appreciate all the work that has been done in researching this issue.  Usually when extraordinary results are reported, like this, I take to heart the axiom, "If it's too good to be true, then it must not be true."

I hope this experience will help us as practitioners read reports with a more critical eye.  I also hope that future researchers will be careful in how they conduct the research, analyze the results, and report their findings.  All too often, good research is damaged by poor analysis or poor reporting.  Just do not stop doing research.

Steven Ewert
Fresno Adult School
ABE/GED Instructor

-----Original Message-----
From: familyliteracy-bounces at nifl.gov [mailto:familyliteracy-bounces at nifl.gov] On Behalf Of tsticht at znet.com
Sent: Tuesday, June 24, 2008 2:41 PM
To: familyliteracy at nifl.gov
Subject: [FamilyLiteracy 1132] Re:Family literacy findings released todayshow one grade level gain for every 10-13 hours of instruction  

Lorri and others:  We have learned a bit more about the NCFL study for which
the Press Release states: Quote: "In year one, there was a one grade-level
gain for every 10 hours of reading instruction. In year two, there was the
same gain for every 12.5 hours." End Quote. Now Chris Schatschneider has
added new information for the list and said: Quote "The claim was made that
a one grade level gain was made for every 10-13 hours of instruction. This
claim is based upon looking at students who had 1 hour to 78 hours of
instruction (with a mean of 27 hours of instruction). Additionally, over
80% of the students received more that 10 hours of instruction." End Quote

Earlier we learned from an NCFL research brief about this study that for 82
Year 1 and Year 2 students combined, they went from an average pre-test
score of 4.96 to an average post-test score of 5.89 for a n average gain of
1.03 years. Using the new information from Chris, that the mean number of
instructional hours was 27 hours, we can calculate the average gain rate
per hour of instruction as 1.03/27=0.038 years gain per hour. Rounding we
get .04 years gain per hour of instruction. When multiplied by 10 we get
0.4 years (i.e., 4 months) of gain in 10 hours. In 13 hours we get 0.5
years (i.e., 5 months) gain. In neither case do we get 1 year of gain for
every 10-13 hours of instruction.

Chris has also indicated that Quote: "The idea that grade level is not on an
interval
scale is most likely true." End Quote. What this indicates is that computing
these sorts of gain rates per hour of instruction and then multiplying the
hourly gain rates by 10, 20, 30, etc hours to indicate how much gain one
would get in 10, 20, or 30 hours of instruction is a highly questionable
procedure. It assumes a constant amount of learning per hour.

Anyone who has taught adult literacy will know that the amount of learning
needed to get from one TABE score or another does not occur for equal raw
scores. For instance, on the TABE 7 survey level E for Reading if a student
gets a pre-test score of 0 out of 25 items correct he or she gets a 0.0
reading grade level score. If they later improve enough to go from getting
0 points correct to getting 6 points correct on the post-test, they still
get a 0.0 grade level score, indicating they learned nothing.

But if the student had scored a raw score on the pre-test of 6 correct, and
gotten a grade level score of 0.0, and then scored 2 more raw scores (8) on
the TABE at the post-test, they would get a grade level score of 1.4 years
(14 months). So a gain of 6 raw score points in the first case indicates no
learning, while a gain of 2 points higher up the scale indicates 14 months
worth of learning!

These types of problems with the creation of scales for measuring learning
gains using reading grade levels are well known to both researchers and
practitioners in both children's and adult's reading acquisition. Further,
anyone experienced in   teaching either children in the K-12 system or
adults in the Adult Education and Literacy System, or in Even Start or
other forms of family literacy programs, knows that learning does not
proceed in equal amounts from one hour to the next. It is therefore not a
valid representation of learning to calculate the gain per hour and then
multiple it by 10, 20, 30, etc hours of instruction and claim that that
validly represents the way adults did learn, and by implication, will learn
in another class if one just follows the instructional procedures used in
the NCFL research.

As I read the NCFL research description, with the average gain of 1.03 years
in an average of 27 hours, this seems like the sorts of gains I have seen in
numerous other reports many of which are summarized in Part 3 of  Adult
Literacy in the United States: A compendium of quantitative data and
interpretive comments by Sticht & Armstrong. You can download a copy of
this report at the www.nald.ca web site in the library pages. You can also
get to it by googling it.

Page 146 of the foregoing report shows pre-post-test data and hours of
instruction for 23 programs in Illinois in 1988, one of which made 1.5
years gain in 25.5 hours, another made 1.0 years gain in 23.8 hours, and
there are some that did not make as much gain in more hours. Another set of
16 programs with pre-post-test data are given on page 150 with one making
1.2 years gain in 32 hours, another making 1.5 years gain in 49 hours, and
others taking many more hours to make less gain.

All told, then, it seems as though the NCFL research produced yeoman results
in pre-post-test gains, but were reported in the Press Release and Research
Brief with an excess of exuberance. For example, the Press Release states:
Quote: "These rapid gains are extraordinary for adult education programs.
They show what is possible with strategic instruction based on research.
When instructors are trained to analyze the exact literacy skills that are
missing in each individual and then helped to deliver instructional
strategies that are laser-focused, the results are extremely impressive."
End Quote

In addition to being careful and thoughtful in the design and conduct of
research on adult literacy education, it is also necessary to be accurate
in reporting and disseminating the findings of the research. Readers of the
research need to exercise critical judgment in evaluating all facets of the
research: design, conduct, reporting, and dissemination.

Tom Sticht

----------------------------------------------------
National Institute for Literacy
Family Literacy mailing list
FamilyLiteracy at nifl.gov
To unsubscribe or change your subscription settings, please go to http://www.nifl.gov/mailman/listinfo/familyliteracy
Email delivered to sewert at fas.edu