Return-Path: <nifl-assessment@literacy.nifl.gov> Received: from literacy (localhost [127.0.0.1]) by literacy.nifl.gov (8.10.2/8.10.2) with SMTP id fABEuF015414; Sun, 11 Nov 2001 09:56:15 -0500 (EST) Date: Sun, 11 Nov 2001 09:56:15 -0500 (EST) Message-Id: <001001c16ac0$d8fb56a0$34a42a3f@computer> Errors-To: listowner@literacy.nifl.gov Reply-To: nifl-assessment@literacy.nifl.gov Originator: nifl-assessment@literacy.nifl.gov Sender: nifl-assessment@literacy.nifl.gov Precedence: bulk From: "gdemetrion" <gdemetrion@msn.com> To: Multiple recipients of list <nifl-assessment@literacy.nifl.gov> Subject: [NIFL-ASSESSMENT:44] Re: question X-Listprocessor-Version: 6.0c -- ListProcessor by Anastasios Kotsikonas X-Mailer: Microsoft Outlook Express 5.50.4133.2400 Content-Transfer-Encoding: 7bit Content-Type: text/plain; Status: O Content-Length: 13784 Lines: 249 Nancy, Dianna, and others: (note, this is a bit long) This is what one student shared with us who is at a very low level reading ability, and who has made little or no progress on standardized levels, and has certainly not progressed on any NRS levels: "I like to do a lot of things with my grandkids. Every time I do stuff, they 're right there learning it. Then my granddaughter ask me a question, like "what is it. She's like, "grandma, what is this word?" I'm like "cat." She like, "how can I spell that?" "C-a-t, cat." And then she'll write it five times, then I have to spell it. That's how I get taught the learning, then I'll spell it myself. And I help her out more than me. Most everything I be doing, she catch on with me. We about at the same level. Yeah, I be helping a lot of people out. That's my goal." The mission statement of our agency is the following: ..to create opportunities for Hartford area adults to learn to read, write and speak English; to gain ways to represent themselves; and expand their contributions to their families and communities." Sometimes the result can be, at least in part, represented through progress on standardized tests. At many other times it cannot. While this individual's progress on increasing her reading ability in any standardized way is negligible, what she has gained by participating in the program has had a positive effect on her life and her family. Her experience could be looked through at increasing levels of depth, which would even disclose more about the various impacts of adult literacy within the lives of individuals who might not make such progress on standardized tests. Thus, such qualitative information can be documented. Like you, Nancy, and many others, I could easily organize several hundred examples of the ways in which individuals draw on literacy and the various meanings that they attach to it. Combine that with a degree of analysis and with cross program information and you have some decent documentation that can illuminate a great deal about the many ways that adult literacy enhances the lives of individuals, as well as pointing out the limitations of adult literacy as a single source of life influence. However, and here's the rub, that documentation comes in the form of narration, analysis, and interpretation, what certain anthropologists refer to as "thick description." From certain research traditions, this form of scholarship, referred as "ethnography," is perfectly legitimate and discloses insights and information that would be difficult from research traditions primarily based on a quantitative mode. The tragedy and, if you will, or to state it more strongly, the farce, is that there is legitimacy in both modes of scholarship, though they serve different purposes, but only the quantitative is legitimized at the level of policy. Broad based quantitative data can provide general information over wide areas of population and geography, which is worth something. Yet, the holy grail quest for *comparability* at least in terms of adult literacy, is skewered for at least three reasons: a) The context of what one is reading plays a significant role in influencing the "level" of reading one can master in any single text. The less one knows or the less one is interested or motivated or can make sense of a particular text, the more one has to rely more exclusively on the text rather than on one's experience and knowledge-based to decipher the text. The opposite is also true. The more one knows and is interested in a text, the more one is able also to draw from personal context in making sense of a particular text. Much contemporary reading theory discusses the interactive process between the text, the context, and the reader. Effective reading requires a coming to terms with all these dimensions. Thus, any effort to attain broad-based comparability based on the levels metaphor is skewered by the relationship of the content of the tests to the personal knowledge and interest of the individuals taking the test. Then there is the matter of the artificial nature of a paper and pencil test in the first place to the ways people actually use text in real-life situations. This point is well documented in academic literature on literacy, but does not seem to have informed the research that went into the National Reporting System. But perhaps I'm wrong on that. b) One wonders what it is that is being compared, given the different, largely standardized tests that are allowed for NRS reporting. You're using TABE. We're using CASAS. They are looking for different things. Moreover, CASAS is based on a specific curriculum focus stemming from the five categories of "survival skills" stemming from the 1975 Adult Performance Level (APL) study. Most of the other tests purport to assess reading ability. Should EFF performance indicator be accepted by the NRS, they will be based yet at on other aspects of the reading process including such performance indicators as "determine reading purpose" and "analyze the information and reflect on its underlying meaning." Looking at these three forms of assessment alone, one might surmise that drawing objectively-based, uniform like comparability that then are reduced to NRS levels, is problematic on its face. This argument has also been made ad nausea on various listservs, among practitioners, and in the scholarly literature. Yet, it is ignored, minimized, ignored, and/or discounted by the policy sector. Why is that so? Inquiring minds want to know. c) In various articles, Regie Stites speaks about content standards, performance standards, and opportunity to learn (OTL)standards. On his reading, valid assessment requires alignment of these three dimensions. To push that argument a bit further, one might say that unless they are aligned, assessment can not achieve the critical criteria to which some research traditions point of "validity" and "reliability." When one draws on the BEST, TABE, CASAS or EFF performance indicators (still under development), or whatever, the only basis for their reliability is their congruence with what is being taught. We won't even get into the OTL standards in this message, what one researcher refers to as being sacrificed to "the realities of American politics." Yet, this standard is so fundamental, that its lack of being factored in, makes the holy grail quest for objectivity problematic on its face. See the following NIFL-FOB message for additional discussion. http://literacy.nifl.gov/nifl-fobasics/1999/0200.html The issue I focus on here is the wide gulf between the many contexts that give shape to the content of adult literacy education and the standardization implied in such testing. That is, the gap between content standards and performance standards. These are not at all addressed in the NRS, which is simply drawing on the performance indicators of the various tests in order to determine reading levels, through which one then infers comparability. Such an arrangement smacks more of political accommodation in the effort to come to terms with the siren call of accountability than it does of intellectual legitimacy in establishing a coherent assessment system based on logic, evidence, research, and best practice. These issues have also been widely discussed on the lists, among practitioners, and in the scholarly literature, much of which does not inform current policy perspectives. In my view, one should be concerned about this. I do believe there are ways out of this morass that point to a need to change the *metaphor* on how assessment is determined from that of quantification to that of narration as a founding source of logic that then would need to be buttressed by other information. It would also require a coming to term with sampling and attaining multi-measures among a selective group of students, including in-depth interviews and other site-based qualitative information, which could be factored into a broader report that might actually be useful both to programs and to state and national ABE support agency. This would also require a coming to terms with *ethnography* as a legitimate research tradition that would fit in well with the approach to assessment which I am briefly indicating here. If the need were no longer for standardized, uniform, quantitative information on every single student, but a focus more on program evaluation and broad-based comparability that accounts for differences, then such a research tradition would fit in well. It would better help in getting a handle on what's going on in the field of adult literacy and what can be done to strengthen the system. But that would require a more sustained policy commitment to the public value of adult literacy and also a change in the metaphor, which determines such value. To push this a bit further, I would suggest that one of the core problems is the capitalistic metaphor itself (yes, and it is a metaphor), "return on investment," which reinforces a somewhat narrow cost-benefits utilitarian, quantitatively-driven analysis that can be discretely and precisely measured, and would discount the value of the story with which I opened this message. Hey, I might shed a tear, but it doesn't count in the real world. That sentiment, my friends, is one of the core problems that profoundly limits what counts. If that is the case, in fact, while we want to hear such stories at our conferences, we know in the hard world of policy really, they matter for absolutely nothing, which translates into an utter marginality of anything to do with literacy as expressing a human face that cannot be aggregated into a comparability chart. In Rethinking Literacy Education (1997), Quigley has written incisively about these matters. I believe it is particularly the task of community-based literacy programs to play a significant role in telling that story as well as the responsibility of state and national ABE directors to provide full scope for such programs and not try to place the entire field under one tent, the paradigm that best supports state mandated ABE and GED programs. There are many tents throughout the land of literacy. We need a national reporting system that better takes the diversity of the field into account. One size does not fit all and neither do the 6 NRS levels. I could go on, and I hadn't intended to write this long. I guess I just got carried away. George Demetrion ----- Original Message ----- From: <Nashansen@aol.com> To: "Multiple recipients of list" <nifl-assessment@literacy.nifl.gov> Sent: Sunday, November 11, 2001 1:36 AM Subject: [NIFL-ASSESSMENT:43] Re: question > In a message dated 11/07/2001 6:27:04 PM Central Standard Time, > dbaycich@archon.educ.kent.edu writes: > > << I've been thinking about the respones I received from my questions about > alternative assessments (I'm a slow thinker). > I better understand the reluctance to use alternative assessments when the > standardized tests are the ones that "matter". Do any of you think there is > a way to convince the folks who want the numbers that they are not getting > the whole picture of what we do? > Do you think the numbers game will change with the move toward EFF? > I'm interested to hear your thoughts. >> > > > Dear Dianna, > > I also am interested and a very slow thinker .... one who contemplates and > does the best she can to meet the needs of others. > > I have been lurking at this listserv hoping answers will arise because I am > really struggling with the assessment requirement tied to any future funding. > I want to know what it is that the "powers that be" are attempting to > measure when it comes to TABE testing someone with as limited literacy skills > as I see in the adult population I serve in the Sioux Falls Area Literacy > Council in South Dakota. I have even tried to advocate against the timed > testing and using such standardized tests with our new AEL Director within > the S.D. Dept. of Labor and really have had a deaf ear turned. (She gives no > clear cut answer to the above question either.) > > You asked if the numbers game will change with the move toward EFF: I feel > it already has become more of an emphasis. I don't quite know if it was the > switch to EFF or the switch from Dept. of Educ. to Dept. of Labor > allocations. The emphasis appeared to me to take an immediate sharp turn > toward employment as a primary focus and unfortunately for *my* learners, > that's truly not the case for their enrollment. Their goal in enrolling also > wasn't primarily prep for GED acquisition *either* but at least there was > some fluidity with the DOE. > > It frightens me. I think this turn of events will eventually pull the > pillars right from under *my* program ... don't know about others and > anxiously await input as to whether they feel as strongly as do I. The > purpose for learning to read is so individualized that each of the adults > with whom I match a volunteer will express diverse reasons. They might say > "I just had to *do* this for MySelf!" or "I lost my job because there was no > other job in the workplace for me when I was injured and didn't have enough > literacy skills to transfer to a position requiring reading, writing and > spelling." Or maybe that prospective learner hasn't even gotten to first > base when it comes to a job search. Another learner might be doing it > because they want to be able to write a letter home, read a preschooler a > book, find a street in their community. I could go on, but won't. > > I am interested in further response as well. > > Nancy Hansen > Sioux Falls Area Literacy Council > Sioux Falls, SD > sfliteracy@mcleodusa.net >
This archive was generated by hypermail 2b30 : Fri Jan 18 2002 - 11:33:24 EST