Colleagues, 

Below Tom discusses 3 areas of assessing adult literacy:  at the
national level, at the program level, and at the individual level.  Tom
notes that all these types of assessments are in use today to determine
the effectiveness of our delivery systems and the quality of our
programming, but he also notes that the results of these efforts still
remain unclear; we need more research in these areas.  

What are your thoughts on Tom's discussion?  Many of us do believe that
more research is needed, but without the Adult Literacy research arm
(NCSALL), how can we proceed with studies and research that can help
inform the future of our work?  

Toward the end of his discussion, Tom notes that many of us are looking
at what is happening in the classroom (formative assessment) to
understand how what happens at that level can inform and shape what
happens on the broader levels, but making these connections is neither
clear nor simple.  

What are your thoughts on this?  Do you think we should start with the
classroom and branch out to the broader levels, or is this unrealistic?


I will be interested to see if any of our panel guests in next week's
discussion on Using Data for Program Improvement touches on any of the
points that Tom makes in this post.

Thanks,

Marie Cora
Assessment Discussion List Moderator

**********

April 12, 2007

Strategies for Evaluating the Effectiveness of Adult Literacy Programs:
An Update From 1990

Tom Sticht
International Consultant in Adult Education

Over a decade and a half ago, in April of 1990,  I participated in a
National Conference on State Literacy Initiatives in Harrisburg,
Pennsylvania. At the conference, which was co-sponsored by the National
Governor's Association, the U. S. Department of Education, Project
Literacy
U.S. and three organizations in Pennsylvania, I was asked to present a
plenary session on the topic of "Strategies for Evaluating Effectiveness
of
Programs."

In my presentation I discussed three areas of assessment of adult
literacy.
The first was at the national level, such as what we have today in the
National Assessment of Adult Literacy (NAAL). These types of assessments
are aimed at giving us an idea of the literacy skills of adults
regardless
of how or where they may have acquired these skills.

A second arena for assessment is in the evaluation of programs. Here, we
evaluate programs to see if the programs are meeting standards, to see
if
they are maintaining their quality, and to see if programs are improving
in
their effectiveness. My presentation focused on this second area of
assessment of literacy.

The third arena for assessment that I discussed was the assessment of
individuals for instructional decision making. I noted that individuals
are
assessed to see where they should be placed in an instructional program,
to
diagnose their specific needs, to determine instructional materials and
methods, and to certify that learning has taken place.

Today, all of these types of assessment of adult literacy skills are
widespread. At the national level since 1990 we have had two national
assessments of adult literacy, the National Adult Literacy Survey (NALS)
of
1993 and a decade later the National Assessment of Adult Literacy (NAAL)
of
2003. Despite the repeated reporting of 30, 60, or 90 million adults
with
literacy skills judged inadequate by the U. S. Department of Education
and
the National Governor's Association in both 1993 and 2003, some 95
percent
of adults reported that they did not think they have a literacy problem
and
federal policies in the management of adult literacy programs appear to
have
lead to a loss of some one million enrollments in programs since 1993.
Additionally, investments in the Adult Education and Literacy System
(AELS)
today barely exceed $200 per enrollee, less than 3 percent of the $6500
provided for each child in the federal Head Start program. All this
suggest
that we need to rethink the nature of our national assessment of adult
literacy and what our national and state policies ought to be in the
light
of what new assessments reveal.

Today, in keeping with my second use of assessment, that is, to see if
programs are meeting standards, to see if they are maintaining their
quality, and to see if programs are improving in their effectiveness,
there
is a National Reporting System (NRS) that collects data on learning
gains
and several outcomes such as secondary degrees/certificates awarded,
transition into further education/training, employment and so forth.
However, there are not yet national standards for quality nor comparable
national assessment tools that would make valid comparisons of how well
different programs and different states are performing in both
maintaining
acceptable quality standards and monitoring their programs so that drops
in
performance can be corrected and steps can be taken to improve
performance.
Further, the NRS data on learning do not include pre- and post-test
scores
so that  gains in learning can be estimated more accurately. Resources
also
need to be made available that would permit the numerous programs to
return
that dropped out of the AELS, taking some million enrollments with them,
when the NRS demands for data collection were imposed across the nation.

The third arena for assessment that I discussed in 1990 was the
assessment
of individuals for instructional decision making. Today there is a
wide-ranging discourse on various discussion lists on formative
assessment
for instructional decision making, assessment for learning disabilities,
diagnostics in assessing components of reading ability, and other
aspects
of instructional decision making. Whether or not this discussion and
dialogue is actually improving instructional practices in the AELS is
not
known to me. However, if classroom decision making is supposed to lead
to
improvements in learning which are then captured in program assessments
using pre- and post-test data on standardized tests, as the present NRS
calls for, and if the latter is to lead to improved scores by adults on
the
national assessments, then the recent NAAL data suggest that any
improvements in the AELS classrooms over the decade from 1993 to 2003
did
not bubble-up to the national level. There were as many adults in the
lowest level of literacy in 2003 as there were in 1993.

To repeat my conclusion from the 1990 conference, "As pedantic as it may
sound, more research is needed to develop strategies for the assessment
of
programs that can improve adult literacy in the coming decade."

Thomas G. Sticht
International Consultant in Adult Education
2062 Valley View Blvd.
El Cajon, CA 92019-2059
Tel/fax: (619) 444-9133
Email: tsticht at aznet.net