Return-Path: <nifl-workplace@literacy.nifl.gov> Received: from literacy (localhost [127.0.0.1]) by literacy.nifl.gov (8.10.2/8.10.2) with SMTP id f72KP6f08354; Thu, 2 Aug 2001 16:25:07 -0400 (EDT) Date: Thu, 2 Aug 2001 16:25:07 -0400 (EDT) Message-Id: <p0500190db78f66002e91@[146.186.96.31]> Errors-To: listowner@literacy.nifl.gov Reply-To: nifl-workplace@literacy.nifl.gov Originator: nifl-workplace@literacy.nifl.gov Sender: nifl-workplace@literacy.nifl.gov Precedence: bulk From: Barb Van Horn <blv1@psu.edu> To: Multiple recipients of list <nifl-workplace@literacy.nifl.gov> Subject: [NIFL-WORKPLACE:259] X-Listprocessor-Version: 6.0c -- ListProcessor by Anastasios Kotsikonas Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="iso-8859-1" ; format="flowed" Status: O Content-Length: 8682 Lines: 158 Dear subscribers, Tom Sticht sent the following to me a week or so ago. It's being posted a little later than I expected due to my vacation and work travel schedule, but I think you'll find it interesting... Barb Van Horn Co-moderator NIFL-workplace __________________ Tom said: Based on the Research Note below that I posted on the NLA list July 10, 2001, Jay Mathews, education columnist for the Washington Post, published an article Tuesday, July 17, 2001 entitled "Adult Illiteracy, Rewritten" in which Andrew Kolstad, former director of the National Adult Literacy Survey reports that the NALS used "the wrong "response probability" even after other federal researchers had concluded that that would exaggerate the severity of [adult literacy] problems." The full text of the Washington Post article can be found at http://www.washingtonpost.com/wp-dyn/articles/A6155-2001Jul16.html Research Note 10 July 2001 Has the National Adult Literacy Survey (NALS) Defamed the Competence of America's Labor Force? Thomas G. Sticht International Consultant in Adult Literacy In 1993 the National Center for Education Statistics (NCES) released the first report on the National Adult Literacy Survey (NALS) of 1992. In the report, the developers assigned adults to five different literacy levels - 1 (low) through 5 (high). To qualify to be at a given level, an adult had to have an 80 percent (p=.80) chance of being able to perform the average task at the given level. Following this decision rule, some 20 percent of adults were placed in Level 1, while 27 percent were placed in Level 2 (prose scale). This led to the quote in many newspapers that "half of America's adults are functionally illiterate!" Political cartoonists made fun of America's "illiterate" adults, and these sentiments were subsequently expressed internationally by leaders in Japan that defamed the skills of the American labor force. In the NALS scales, adults with skills of 200 were assigned to Level 1, because they could do 80% of the average tasks at that level, however, they could also actually do 45% of the tasks at Level 2, 25% of those at Level 3, and even 15% (one in six) of those in Level 5. Adults with scores of 250 were assigned to Level 2, and it was implied that they could not perform more difficult tasks, even though they could do half (50%) of the tasks at Level 3, and one in five (20%) of the tasks at Level 5, the highest level of difficulty. But by being called Level 2 adults, all competence above that Level was (at least implicitly) denied to them by reporters and policymakers. An interesting and somewhat puzzling finding of the NALS was that most adults in Levels 1 and 2 reported that they could read and write English "well" or "very well." That is, contrary to what the test developers concluded based on the literacy tests, the adults who were called poorly literate did not think that their literacy skills were very poor. NALS Validity Challenged in New NCES Technical Report Now, almost a decade after the publication of the first NALS report, a final technical report on the NALS has been published by NCES entitled: Technical Report and Data File User’s Manual for the 1992 National Adult Literacy Survey, NCES 2001-457, January 2001. It is a huge report with some 15 chapters and 18 appendices covering more than 500 pages of mostly detailed technical data. It is not comfortable reading. Still, an astonishing discovery is made after prodding through some 13 chapters of technical information on the NALS. And that discovery is Chapter 14. After some 347 pages of information about the NALS, its theoretical foundation, construction, administration, and interpretation, Chapter 14, written by Andrew Kolstad, the original project director for the NALS at NCES, systematically undermines the entire test and questions its construct validity, that is, the question of just what it is that the test measures, its standards validity, that is, the validity of the 80 percent probability standard that was used to assign people to the five literacy levels, and the use validity, that is, the validity of the NALS for accurately identifying adults at risk for poor literacy skills. Regarding the construct validity of the NALS, Chapter 14 reports that one of the major predictors of success on the NALS, "plausibility of distractors," was statistically significant when the 80 percent probability level of success was used, but when lower success rates were used to scale the test items, that variable became non-significant, while a "readability" variable which was not significant at the 80 percent probability level became a significant predictor of performance on the tests (p. 360). Thus, the theoretical components describing what the tests measure changed, not as a function of the nature of the test items, but as a function of the change in response probabilities used to scale the items for difficulty. This challenges the theoretical construct of just what it is that the NALS measures. Regarding the standards validity, Chapter 14 provides extensive analyses showing the results of using a variety of different response probabilities (RP) from the 80 percent standard used on the NALS on the likelihood of saying that adults can’t do various literacy tasks when in fact they can (false negatives), or saying that they can do literacy tasks when in fact they can’t (false positives). In a conclusion that challenges the accuracy and hence the validity of the 80 percent standard used by the NALS, Chapter 14 concludes that the RP that produces the least errors in saying people can’t or can perform literacy tasks is an RP of .50. The chapter states, " The value of a 50 percent response probability is that it strikes an even balance between false positives and false negatives. With this criterion, one can have the same degree of confidence in statements about what adults can’t do as in statements about what adults can do. Mapping items in this way both provides a balance between false positives and false negative claims about adult performance and minimizes the total misclassification error." (p. 369) In short, one of the reasons why adults who were classified as poorly literate and placed in levels 1 or 2 of the NALS thought they were fairly literate was that they were likely classified incorrectly by the use of the 80 percent RP standard. They could actually perform many tasks that the NALS developers said they could not. Finally, regarding the use validity of the NALS, Chapter 14 states that "Many users of adult literacy survey findings want to focus on the lack of facility with printed and written material and to report on what adults can’t do. To be sure that adults are unable to perform the literacy tasks, survey analysts might need to use a correspondingly low response probability criterion, such as .20 percent. Such a criterion would assure that adults who do not reach these levels of consistency in their responses have a very low chance of success with the tasks." (p. 368) Using an RP of .20 and keeping the same cut points for the five levels of literacy would reduce the percentage of adults declared poorly literate and assigned to literacy level 1 on the prose scale from 20 percent to 5 percent and from level 2 the change would be from 27 percent to 5 percent. Thus, using the .20 RP, only some 10 percent of adults would have been declared as poorly literate. Using the .50 RP would reduce the percentages of adults below level 3 from 47 to 22 percent, less than half that reported by the NALS developers using the RP .80 standard of performance. All this challenges the use validity of the NALS data as reported, which had the negative consequences of the defaming of America’s labor force by both U.S. newspapers and foreign governments. There Needs to Be a National Debate on the Literacy Standards That Adults Should Meet Question: How many functionally illiterate adults are there in the United States? Wag's Answer: As many as you would like! This new technical report by NCES calls into question the entire validity and hence the meaning of the NALS (and other tests and studies based on the NALS methodology, such as the International Adult Literacy Survey). That this major assessment can throw little light on how many adults are at risk for their literacy skills should command a major dialogue and investigation into national adult literacy assessments. This dialogue should happen before another 10 to 15 million dollars are spent in a follow-up survey that may be equally invalid and lead to further defamatory practices by reporters or government officials here and abroad.
This archive was generated by hypermail 2b30 : Fri Jan 18 2002 - 11:28:15 EST