[NIFL-ASSESSMENT:570] Delusion of Accountability in Adult Education

marie.cora@hotspurpartners.com

Date: Mon, 24 May 2004 14:22:56 -0400 (EDT)
Message-Id: <001701c441bc$f69455b0$0302a8c0@frodo>
Errors-To: listowner@literacy.nifl.gov
Reply-To: nifl-assessment@literacy.nifl.gov
Originator: nifl-assessment@literacy.nifl.gov
Sender: nifl-assessment@literacy.nifl.gov
Precedence: bulk
From: "Marie Cora" <marie.cora@hotspurpartners.com>
To: Multiple recipients of list <nifl-assessment@literacy.nifl.gov>
Subject: [NIFL-ASSESSMENT:570] Delusion of Accountability in Adult Education
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
Status: O
Content-Length: 13921
Lines: 318

Hello everyone,

Tom, thanks for this Research Note - George, thanks for forwarding it to
the Assessment List.

I wanted to provide a bit of further info based on Tom's Note, and ask a
few questions for us to ponder.

Much of what Tom is focusing on in the report has to do with setting
benchmarks, which is a term that is being used more and more lately when
discussing how we know that we (or our students) have reached particular
goals or education levels.  

First, here are a few definitions of benchmark or related terms (which I
got from the LINCS Assessment Collection Glossary:
http://literacy.kent.edu/Midwest/assessment/glossary.html)

Benchmark - A detailed description of a specific level of student
performance expected of students at particular ages, grades, or
development levels. Benchmarks are often represented by samples of
student work. A set of benchmarks can be used as "checkpoints" to
monitor progress toward meeting performance goals within and across
grade levels. In ABE, SPLs (Student Performance Levels) are examples of
benchmarks; targets for instruction.

Benchmark tasks - Pieces of student work selected by a group of lead
teachers as exemplifying a certain score level. (See also Anchor Test.)

Benchmarking - Comparing performances of people on the same task; raters
use "anchors" to score student work, usually comparing the student
performance to the "anchor"; benchmarking is a common practice in the
business world. (See also Anchor Test.)

Tom notes part of what the law states or implies is that standardized
assessment (I'm going to interject here that Tom is referring to
commercially produced tests:  I am in the camp that believes you can
standardize any type of assessment, if given adequate time and resources
- remember those questions I posed to you a couple weeks ago!?) is
objective and therefore better as an assessment than teacher judgment.
Questions:  if you do not adhere to the basic tenets of administering a
standardized test AND the requirements particular to that test, then is
it still objective?  If you answer no, then the answer is that there is
no choice but to follow the letter of the tenets exactly, and is that
humanly possible?  500 (150?) years ago, a lot of the way people taught
and learned was via mentorship or apprenticing:  was that not based on
teacher judgment?  Were blacksmiths, bakers, seamstresses, carpenters,
doctors, lawyers, and new teachers of that time not really very good at
their profession because their instructor decided when they were
finished with their study?  

Our new system also, as Tom notes, places very little if any value on
what the adult student herself has to say about the experience.  We have
had some conversation here about capturing those harder to measure
pieces, like self-esteem or appreciation for the classroom.  Teachers:
if you receive back a TABE with a high mark on it, I'm sure you are very
pleased for your student; if you hear that student tell you about the
moment they recognized their own name in print, the time they wrote a
note to their kid's teacher, or when he finally approached the boss for
a raise, I've no doubt you are also really pleased.  Which means more?
Which tells you more?

What are both these pieces good for?  How does the use of commercial
test data help you and the student?  How do anecdotal stories help you
and the student?  Is it one or the other, or do we need to find a way
that both these pieces can be accounted for in our and our students'
work? 

Tom suggests toward the end of the Note that engaging groups of students
and teachers in developing the answers and tools to some of these
questions might be a good place to focus.  What do you think?  Is anyone
doing anything like that on a state or local level?

In his reply, George suggests that we need other ways of representing
literacy to the public in order to shift the mindset that the only way
to show success is via an "objective" test.  Can we list out what some
of those other ways might be?  A few authors have written on 'many
literacies' notions - how do we show policy-makers and funders that
there are more and better ways of helping people succeed and of showing
how they did?

(Whew!  Sorry to be long-winded!  This topic really hits the crux for me
I guess!)

So what do YOU think?

marie cora
NIFL Assessment List Moderator

From: Thomas Sticht <tsticht@znet.com>
To: <aaace-nla@lists.literacytent.org>
Subject: [AAACE-NLA] Delusion of Accountability in Adult Education
(longer)
Date: Fri, 21 May 2004 16:35:43 -0700 (PDT)

Research Note 				21 May 2004

The Delusion of Accountability in the Adult Education
and Literacy System (AELS) of the United States

Tom Sticht
International Consultant in Adult Education

Definition: Delusion: (1) A false belief or opinion. (2) a persistent
false belief that is a symptom or form of madness-Oxford American
Dictionary.

The Workforce Investment Act, Title 2: Adult Education and Family
Literacy
Act (WIA/AEFLA) was enacted by the U. S. Congress in 1998. It introduced
requirements for accountability including indicators of (1) gains in
learning, (2) placement in, retention in, or completion of,
post-secondary
education, training, unsubsidized employment, or career advancement; and
(3) receipt of a secondary school diploma or a recognized equivalent
[P.L.
105-220 Section 212(b)(2)].

To define and implement the accountability requirements of the WIA/AEFLA
the U. S. Department of Education established the National Reporting
System (NRS). This system collects data about each of the three
categories
of accountability indicators from the states, consolidates it, and
reports
it in an annual report to Congress. The most recent report is:   "U. S.
Department of Education (2003). The Adult Education and Family Literacy
Act: Program Year 2001-2002: Report to Congress on State Performance.
Office of Vocational and Adult Education."

Reading this report I was struck by the delusional beliefs it
propagates.
First is the belief that the Congress can and should somehow use the
information to hold states and local programs accountable for spending
in
a responsible way the funds that are distributed in the State Grants.
But
this is not possible nor desirable because the information is almost
completely useless. It is based on totally arbitrary '"benchmarks" that
have been set at different levels, using different methods, with
different
indicators at the state and federal levels. For instance. to provide
information to Congress about gains in learning, the NRS constructed six
levels of learning for adult basic and secondary education and six for
English Language learning. States are required to report "Percent of
enrolled adults who acquired the level of basic skills needed to
complete
at least one education level (minimum Grade Level Equivalent - 2
years)."
(p. 11).

The report presents a graph on page 11 showing that in Program Year (PY)
2000-2001 36 percent of adults enrolled in ABE/ASE acquired the skills
needed to complete at least one education level. Then in PY 2001-2002
this
rose to 37 percent, but was below the federal Education Department's
(ED)
performance goal of 40 percent.

However, there is no rationale given in the report as to why the federal
ED goal was set at 40 percent achieving the skills to move up one or
more
levels nor is there any indication of what the different state goals
were,
why they were set that way, or how progress was measured in each state.
However, we are told that different states used different tests, with
differing amounts of time between pre- and post-tests to assess growth
in
learning.  But we are not told how many students went from just a raw
score point or two below one level to just barely above the next level,
a
result that could occur from the unreliability of the tests used and not
from learning at all.  Similar problems hold for the data presented for
English Language acquisition.

The report cautions that "The comparison of educational functioning
levels
and level gains across states is thus complicated by this lack of
comparability." (p. 6). But this is a gross understatement, because in
fact this "lack of comparability" in measurement tools and their
administration, coupled with the "lack of comparability" in the methods
of
setting benchmark goals in each state and at the federal level, renders
these data totally meaningless and useless to Congress (or anyone else
for
that matter)  in deciding whether or not states are using their State
Grant funds responsibly and productively. In fact, if any member of
Congress or their staff members, or anyone else did use these data for
holding a state accountable for their State Grant funds then they did so
inappropriately.

The delusional nature of the beliefs about the NRS accountability system
is illustrated by statements in the PY2001-2002 report that accept the
faults identified above  while disparaging the judgments of teachers.
One
statement says that "Assessment systems have greatly improved, with more
consistent and widespread use of standardized, psychometrically sound
assessments and abandonment of subjective assessments or teacher
judgments
that do not accurately measure student learning."(pp. 8-9) This
indicates
the unsubstantiated belief that the standardized assessments are
accurate
and valid measures of student learning while teacher judgments are not.
No
evidence is cited in support of this belief nor is any method suggested
as
to just how one might "accurately measure student learning."

Another delusional belief regarding the NRS stated in the report says
that
"Never before have states and local programs had the ability to make
data-driven decisions to help them design more effective programs and
meet
students' needs."(p.9) This is an astonishing statement suggesting
ignorance of the more than 35 years of human impact data available for
the
State Grant program that supports the Adult Education and Literacy
System
(AELS) of the United States.

Furthermore, if meeting the needs of learners is indicated in part by
the
number of adults who seek out the AELS for services, then the "old" data
may have been more effective in managing programs than the "new" NRS
data.
This is indicated by the fact that for the years from 1966 to 1998
enrollments in the AELS grew at a rate of almost 100,000 a year. In 1998
there were 4,020,550 enrollments in the AELS. After the WIA/AEFLA of
1998
was passed, which introduced new accountability requirements emphasizing
the use of "objective" methods like those of the NRS, enrollments fell
to
3,616,391 in 1999, to 2,891,895 in 2000, to 2,673,692 in 2001, and then
rose slightly to 2,787,414 in 2002. This represents a decline of
1,233,136
(30.6 percent) enrollments in the first four years of the new
accountability system now operated as the National Reporting System
(NRS).
While it may not represent a very sophisticated approach to program
accountability, it seems clear that if adults do not enroll in the AELS,
it isn't going to help them very much.

A final, and particularly distressing indication of delusional beliefs
regarding the NRS accountability system is indicated by a statement that
deprecates the opinions of the adults in the programs and claims that
"Programs have replaced measures such as self-esteem and student
appreciation of the classes with objective measures of student literacy
gains."(p. 9)

This extraordinary statement dismisses the importance of the values that
students place upon their programs and their improved feelings of worth
due to participation in these programs. It reveals the false belief that
standardized tests are "objective measures of student literacy gains" as
though subjective decisions about what content and procedures go into
the
make-up and administration of the tests have not taken place. Worse yet,
it suggests that such tests are not only more objective than student's
judgments, but also more valid indicators of what has been learned by
students in the programs. But remember, these students are adults, not
children. They are tax payers just like other adults. Who is better
equipped and more responsible than they to hold programs accountable for
meeting their learning needs and for determining whether or not they
have
learned useful knowledge and developed better skills?

If anything, teacher and student judgments might prove a much more
useful
approach than that of the (mis)use of standardized test data for
insuring
that funds for the AELS are meeting student needs for learning.  Rather
than working with experts in measurement, psychometrics, and testing,
accountability for Congress might be better served by engaging adult
learner groups such as VALUE to work on devising methods for letting
teachers and adult students in Congressional districts across the nation
better determine whether or not programs are meeting their needs.

But placing trust in the teachers and adult learners in the thousands of
programs in the AELS would require the overcoming of delusional beliefs
about assessment and accountability. And nothing is more resistant to
change than "a persistent false belief that is a symptom or form of
madness."

Thomas G. Sticht
International Consultant in Adult Education
2062 Valley View Blvd.
El Cajon, CA 92019-2059
Tel/fax: (619) 444-9133
Email: tsticht@aznet.net

_______________________________________________
AAACE-NLA mailing list: AAACE-NLA@lists.literacytent.org
http://lists.literacytent.org/mailman/listinfo/aaace-nla
LiteracyTent: web hosting, news, community and goodies for literacy
http://literacytent.org

--------- End forwarded message ----------
________________________________________________________________________
___
Earn $20 for every new person you bring to Juno Platinum or Juno
SpeedBand. To learn how, go to http://www.juno.com/refer  
--------- End forwarded message ----------