Steve and others,

 

I was also surprised by the seemingly erratic nature of the results in
Table 6 of the CAL study that related hours of instruction to
educational functional gains.  A clearer picture begins to appear when
you do a simple z-test to determine whether the % of completions of
students who had more than 60 hours of instruction is statistically
significant compared to the baseline.  As it turns out there are very
few significant differences in the table, with the exception of
Beginning Literacy students,  among whom t the difference in the % of
completions is statistically significant in comparison to the baseline
after 80 hours of instruction.  Interestingly, there are no significant
results for Low Beg. ESL, High Intermediate ESL, and Advanced ESL.
There is a statistically significant difference in completion rates for
High Beginning ESL after 120 hours of instruction and for Low
Intermediate ESL after 140 or more hours.  Seemingly incomprehensible
drops in the level of completions for some of the groups with more hours
of instruction are very small and probably not statistically
significant.

 

Some of these results may be an artifact of the study's sampling
strategy.  Beginning Literacy ESL shows the most significant improvement
in hours of instruction, and, not surprisingly, the sample size for
those groups are much larger (e.g. 1720 for Beg. Literacy vs. 252 for
Advanced ESL), as a result some of the sub-groups are very small, and
may lead to statistically insignificant results.    For instance, the
sample size for Low Beginning students who received 120-139 hours of
instruction is only 37; as a result, the improvement from 75% to 86% in
completions was rendered statistically insignificant.  With a sample
size of at least 200 that same result would have been statistically
significant.   Clearly, the study needs to have a much larger sample for
each group. 

 

I concur with Steve's observation that level of completions found in
this sample was much higher compared to the national ESL population
(congratulations MA and IL), but it may have also contributed to the
relatively low impact that hours of instruction had on the completion
rates of Low and High Beginning ESL.  Both of these groups started a
very high level of completions (75% and 72%) with less than 60 hours of
instruction.  There just wasn't a lot of room for improvement there.

 

Considering the nature of the study, many of the findings are
inconclusive or "descriptive" as the study calls them, so I am not quite
ready to dismiss the impact of teaching hours on attainment, and I
believe we do have other (better) studies that support this connection.
I hope CAL will produce a study that actually answers some of these
questions.

 

While we are speculating, however, the data in the study does raise some
questions about the requirement that students receive 60 hours of
instruction before taking a  post-test with the BEST Plus.  According to
CAL's own data there is no statistically significant difference in the
completion rates of students receiving less than 60 hours compared to
those in the 60-79 hour range for all functional levels.  In the
meantime many programs across Texas (and across the country I assume)
are no longer able to count completions they would have obtained from
students who leave untested before the 60 hours.  Is there a clear
rationale for this rule?

 

 

 

Eduardo Honold

ehonol at sisd.net

 

________________________________

From: englishlanguage-bounces at nifl.gov
[mailto:englishlanguage-bounces at nifl.gov] On Behalf Of Steve Kaufmann
Sent: Tuesday, January 08, 2008 1:54 PM
To: The Adult English Language Learners Discussion List
Subject: [EnglishLanguage 2042] Re: New Study on NRS Level Gain Using
BESTPlus

 

More hours of classroom instruction, and more intense instruction,
should lead to better results, otherwise why bother? The real question
is how much improvement is required to justify the time and money
expended by learners, teachers and tax-payers. 

1.2 million adults are enrolled in federally funded adult ESL classes in
the United States.  36% of these students attained a measurable
educational level gain after a course of instruction.  In this survey of
6,599 adults, 60% showed improvement. Obviously being in a survey has a
big impact on improvement results!

Almost half (49%) of the ESL learners in the survey were at level 0 and
1 on the SPL scale, i.e. "no ability whatsoever' or "functions minimally
if at all in English." Almost 20% were Low and High Beginner level
learners (2 and 3 on the scale). Level 3 is described as "understands
simple learned phrases, spoken slowly with frequent repetitions". At the
other end of the scale 7% of the adults surveyed were Advanced or level
6 on the  scale, described as " can satisfy most survival needs and
limited social demands." Even the advanced learners were still at a
basic level.


>From the tables in the report,it appears that the biggest factor

affecting grade improvement was not hours of instruction but the level
of the learner. Beginner learners (level 2 and 3 on the scale) improved
the most and were the least affected by the amount of instruction. Of
those Low and High Beginners who had the least amount of instruction (
between 2 and 60 hours) almost 75% still managed to improve, whereas
this only went up to 84% for those who had between 140 and up to 512
hours of instruction, i.e. probably at least 3 times as many hours of
classroom instruction . We are told in the report that 78%, or almost 4
out of 5 of these Low and High Beginner learners improved regardless of
the number of hours of instruction. 

The largest group, those with essentially no English skills(49%), as
well as the most advanced group (7%), showed the lowest level of
improvement, but seemed to benefit the greatest from instruction. The
report does not explain this nor the fact that the rate of improvement
sometimes declines with increased instruction.(see tables) 

Intensity of instruction does not have a great affect on results. The
largest group ( 57%) studied an average of 4.5 hours per week and 61% of
these learners showed measurable improvement on the scale. However 31%
of the survey group had less than 2.8 hours per week of instruction and
yet 56%  still managed to improve. The intense group, roughly 12% of the
learners, studied more than 9.3 hours per week. Despite more than double
the hours of instruction, compared to the middle group, the percentage
of learners with measurable improvement only increased from 61% to 66%.
Again it was the Low and High Beginners who improved the most, with the
least impact from instructional intensity.

To me the conclusion is that class instruction obviously does help but
not as much as is often assumed. Instead, I suspect that what really
matters is what the learner does outside the classroom. As the report
says, an adult ESL learner has limited time to spend, "typically 4 and 8
hours per week". Surely we should focus on finding ways to enable these
learners to create more time for learning. In other words we should find
ways to make it easier and more effective for them to learn outside the
classroom, and to encourage them to do so, instead of trying to justify
bringing them to class. Classroom time does not seem to have  a decisive
impact on their improvement.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.nifl.gov/pipermail/englishlanguage/attachments/20080108/abed8c52/attachment.html