Return-Path: <nifl-aalpd@literacy.nifl.gov> Received: from literacy (localhost [127.0.0.1]) by literacy.nifl.gov (8.10.2/8.10.2) with SMTP id j0AN5tn06654; Mon, 10 Jan 2005 18:05:55 -0500 (EST) Date: Mon, 10 Jan 2005 18:05:55 -0500 (EST) Message-Id: <41E9E9F6@webmail.utk.edu> Errors-To: listowner@literacy.nifl.gov Reply-To: nifl-aalpd@literacy.nifl.gov Originator: nifl-aalpd@literacy.nifl.gov Sender: nifl-aalpd@literacy.nifl.gov Precedence: bulk From: jataylor <jataylor@utk.edu> To: Multiple recipients of list <nifl-aalpd@literacy.nifl.gov> Subject: [NIFL-AALPD:1879] From Tom, Delusion of Accountablity in ABE (longer) X-Listprocessor-Version: 6.0c -- ListProcessor by Anastasios Kotsikonas X-Mailer: Infinite Mobile Delivery (Hydra) SMTP v3.62.01 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="ISO-8859-1" Status: O Content-Length: 8951 Lines: 161 Jackie: Following is a Research Note relevant to the AALPD discussion on holding accountable those who pose requirements for accountability. Tom Sticht Research Note 21 May 2004 The Delusion of Accountability in the Adult Education and Literacy System (AELS) of the United States Tom Sticht International Consultant in Adult Education Definition: Delusion: (1) A false belief or opinion. (2) a persistent false belief that is a symptom or form of madness-Oxford American Dictionary. The Workforce Investment Act, Title 2: Adult Education and Family Literacy Act (WIA/AEFLA) was enacted by the U. S. Congress in 1998. It introduced requirements for accountability including indicators of (1) gains in learning, (2) placement in, retention in, or completion of, post-secondary education, training, unsubsidized employment, or career advancement; and (3) receipt of a secondary school diploma or a recognized equivalent [P.L. 105-220 Section 212(b)(2)]. To define and implement the accountability requirements of the WIA/AEFLA the U. S. Department of Education established the National Reporting System (NRS). This system collects data about each of the three categories of accountability indicators from the states, consolidates it, and reports it in an annual report to Congress. The most recent report is: "U. S. Department of Education (2003). The Adult Education and Family Literacy Act: Program Year 2001-2002: Report to Congress on State Performance. Office of Vocational and Adult Education." Reading this report I was struck by the delusional beliefs it propagates. First is the belief that the Congress can and should somehow use the information to hold states and local programs accountable for spending in a responsible way the funds that are distributed in the State Grants. But this is not possible nor desirable because the information is almost completely useless. It is based on totally arbitrary ‘"benchmarks" that have been set at different levels, using different methods, with different indicators at the state and federal levels. For instance. to provide information to Congress about gains in learning, the NRS constructed six levels of learning for adult basic and secondary education and six for English Language learning. States are required to report "Percent of enrolled adults who acquired the level of basic skills needed to complete at least one education level (minimum Grade Level Equivalent — 2 years)." (p. 11). The report presents a graph on page 11 showing that in Program Year (PY) 2000-2001 36 percent of adults enrolled in ABE/ASE acquired the skills needed to complete at least one education level. Then in PY 2001-2002 this rose to 37 percent, but was below the federal Education Department’s (ED) performance goal of 40 percent. However, there is no rationale given in the report as to why the federal ED goal was set at 40 percent achieving the skills to move up one or more levels nor is there any indication of what the different state goals were, why they were set that way, or how progress was measured in each state. However, we are told that different states used different tests, with differing amounts of time between pre- and post-tests to assess growth in learning. But we are not told how many students went from just a raw score point or two below one level to just barely above the next level, a result that could occur from the unreliability of the tests used and not from learning at all. Similar problems hold for the data presented for English Language acquisition. The report cautions that "The comparison of educational functioning levels and level gains across states is thus complicated by this lack of comparability." (p. 6). But this is a gross understatement, because in fact this "lack of comparability" in measurement tools and their administration, coupled with the "lack of comparability" in the methods of setting benchmark goals in each state and at the federal level, renders these data totally meaningless and useless to Congress (or anyone else for that matter) in deciding whether or not states are using their State Grant funds responsibly and productively. In fact, if any member of Congress or their staff members, or anyone else did use these data for holding a state accountable for their State Grant funds then they did so inappropriately. The delusional nature of the beliefs about the NRS accountability system is illustrated by statements in the PY2001-2002 report that accept the faults identified above while disparaging the judgments of teachers. One statement says that "Assessment systems have greatly improved, with more consistent and widespread use of standardized, psychometrically sound assessments and abandonment of subjective assessments or teacher judgments that do not accurately measure student learning."(pp. 8-9) This indicates the unsubstantiated belief that the standardized assessments are accurate and valid measures of student learning while teacher judgments are not. No evidence is cited in support of this belief nor is any method suggested as to just how one might "accurately measure student learning." Another delusional belief regarding the NRS stated in the report says that "Never before have states and local programs had the ability to make data-driven decisions to help them design more effective programs and meet students’ needs."(p.9) This is an astonishing statement suggesting ignorance of the more than 35 years of human impact data available for the State Grant program that supports the Adult Education and Literacy System (AELS) of the United States. Furthermore, if meeting the needs of learners is indicated in part by the number of adults who seek out the AELS for services, then the "old" data may have been more effective in managing programs than the "new" NRS data. This is indicated by the fact that for the years from 1966 to 1998 enrollments in the AELS grew at a rate of almost 100,000 a year. In 1998 there were 4,020,550 enrollments in the AELS. After the WIA/AEFLA of 1998 was passed, which introduced new accountability requirements emphasizing the use of "objective" methods like those of the NRS, enrollments fell to 3,616,391 in 1999, to 2,891,895 in 2000, to 2,673,692 in 2001, and then rose slightly to 2,787,414 in 2002. This represents a decline of 1,233,136 (30.6 percent) enrollments in the first four years of the new accountability system now operated as the National Reporting System (NRS). While it may not represent a very sophisticated approach to program accountability, it seems clear that if adults do not enroll in the AELS, it isn’t going to help them very much. A final, and particularly distressing indication of delusional beliefs regarding the NRS accountability system is indicated by a statement that deprecates the opinions of the adults in the programs and claims that "Programs have replaced measures such as self-esteem and student appreciation of the classes with objective measures of student literacy gains."(p. 9) This extraordinary statement dismisses the importance of the values that students place upon their programs and their improved feelings of worth due to participation in these programs. It reveals the false belief that standardized tests are "objective measures of student literacy gains" as though subjective decisions about what content and procedures go into the make-up and administration of the tests have not taken place. Worse yet, it suggests that such tests are not only more objective than student’s judgments, but also more valid indicators of what has been learned by students in the programs. But remember, these students are adults, not children. They are tax payers just like other adults. Who is better equipped and more responsible than they to hold programs accountable for meeting their learning needs and for determining whether or not they have learned useful knowledge and developed better skills? If anything, teacher and student judgments might prove a much more useful approach than that of the (mis)use of standardized test data for insuring that funds for the AELS are meeting student needs for learning. Rather than working with experts in measurement, psychometrics, and testing, accountability for Congress might be better served by engaging adult learner groups such as VALUE to work on devising methods for letting teachers and adult students in Congressional districts across the nation better determine whether or not programs are meeting their needs. But placing trust in the teachers and adult learners in the thousands of programs in the AELS would require the overcoming of delusional beliefs about assessment and accountability. And nothing is more resistant to change than "a persistent false belief that is a symptom or form of madness." Thomas G. Sticht International Consultant in Adult Education 2062 Valley View Blvd. El Cajon, CA 92019-2059 Tel/fax: (619) 444-9133 Email: tsticht at aznet.net
This archive was generated by hypermail 2b30 : Mon Oct 31 2005 - 09:48:29 EST