Return-Path: <nifl-assessment@literacy.nifl.gov> Received: from literacy (localhost [127.0.0.1]) by literacy.nifl.gov (8.10.2/8.10.2) with SMTP id fAL1RO010070; Tue, 20 Nov 2001 20:27:24 -0500 (EST) Date: Tue, 20 Nov 2001 20:27:24 -0500 (EST) Message-Id: <001f01c1722b$752e18c0$e966c143@computer> Errors-To: listowner@literacy.nifl.gov Reply-To: nifl-assessment@literacy.nifl.gov Originator: nifl-assessment@literacy.nifl.gov Sender: nifl-assessment@literacy.nifl.gov Precedence: bulk From: "gdemetrion" <gdemetrion@msn.com> To: Multiple recipients of list <nifl-assessment@literacy.nifl.gov> Subject: [NIFL-ASSESSMENT:56] Additional theoretical and political considerations X-Listprocessor-Version: 6.0c -- ListProcessor by Anastasios Kotsikonas X-Mailer: Microsoft Outlook Express 5.50.4133.2400 Content-Transfer-Encoding: 7bit Content-Type: text/plain; Status: O Content-Length: 11565 Lines: 195 Colleagues, I posted this (below) on the NLA about 10 days ago. The specific post was a response to a topic on the value of research to practice and policy. Yet, given this message's emphasis on theoretical and political issues underlying adult literacy assessment, I thought it might have some bearing for this list. Also, I'm hoping that the points I am making (based on a wide stream of "legitimate" scholarship which seldom percolates upward to the policy sector) will be duly considered by the National Academy of Sciences Board on Testing and Assessment as they work toward developing an independent perspective. Note the importance I stress in my NLA posting on the ethnographic research tradition linked to the academic discipline of anthropology. Insight stemming from this tradition is based on a different set of premises than that which characterizes the "harder" social sciences in search of the "objective." Rather than disclosing anything about "objectivity," ethnography is an interpretive human science all the way down. Its findings are still based on evidence, in part through a methodology that is sometimes referred to as "triangulation." Defined literally, this methodology is based on drawing on three distinctive types of evidence about a particular topic. Thus, in evaluating a student's learning, drawing on a triangular methodology, one might look at student work over a period of time, conduct an interview, and consult with the instructor. I'm just being suggestive here. What ethnography seeks in getting at any particular phenomenon is a "thick description," in this case, of how learning takes place; that is, some sense of the development of the learning process, and what such learning means over time as interpreted by the learner. Ethnography would draw on other related disciplines like psychology, sociology, political science, history, etc., to help construct a well rounded picture of what the researcher is attempting to describe. Yet any such description, however sophisticated, remains an interpretation, the findings of which are subject to other evidence and other points of view. What provides a degree of coherency to this is the emergence of research traditions, communities of scholars that develop certain canonical standards that help to give some structure to what may be deemed more or less legitimate scholarship, though such canon-based standards allow for considerable flexibility as new insights perpetually challenge old orthodoxies. To close this introductory note, let me assert that there are some fundamental differences between viewing assessment from a qualitative as opposed to a quantitative perspective. Moreover, unless the qualitative is given a predominant role in helping to define the canon of legitimate assessment accountability, then something is profoundly skewered in the field. Skewered, I say, because it does not provide scope for the importance of human experience in helping to determine what is valid assessment--"valid" as always being defined by particular groups of people, based on particular standards, and for particular purposes. In fact, one could argue that there is nothing "objective" about assessment standards, but that it is interpretation based on selective evidence all the way down. The question is what set of human standards will prevail in any given culture and for what set of reasons? What is gained and what is lost in the process? Can we take more comprehensive and more inclusive approaches in establishing our research traditions in terms of what counts in the matter of adult literacy assessment accountability? I believe this is an important issue for the National Academy of Sciences Board on Testing and Assessment as well as the field as a whole. Are we willing to confront it? George Demetrion To: <nla@lists.literacytent.org> Subject: Re: [NLA] Can research improve policy or practice? Regie, Tom, and others: I'm wondering too, that in this discussion of relevant research, whether we can also examine various research tradition that stem, say, not only from the "harder" social sciences, but those stemming from social philosophy and the humanities. All of these traditions can potentially draw on what may be viewed as the "facts," but it's their interpretation which is often contestable. I, for one, am concerned about the more recent (and perhaps not so recent) thrust to link "research-based" evidence with the imagery of "objectivity" as portrayed in the hard sciences and mathematics (though, in certain respects Einstein deconstructed Newton, and brought to the realm of physics the theory of relativity). Where, for example, does the ethnographic research tradition fit in with what might be considered useful and, from a policy perspective, legitimate research? While ethnography does not purport to be statistically representative, its "thick description" of phenomenon, which includes analysis and interpretation, including in some cases, statistics, brings a depth dimension to whatever is being studied, which is often unavailable in quantitative studies. Recall that Merrifield, Bingman, Hemphill, and deMarrais' (1997) Literacy Language, and Technology in Everyday Life (Teachers College Press) originated as a policy study under the U.S. Congress Office of Technology Assessment p. vii). If such research was funded by the government, then one would presume that its findings would have some impact on policy. And perhaps there was such impact. If so, it would be instructive to learn something of its nature. Recall also that in its final rendition, the book was an in-depth case study of 12 literacy and ESOL students in Appalachia and California. One of the critical theses of the book is the study of "literacy practices" as an indirect variable that interfaces (or perhaps serves as a catalyst) with other contextual factors to effect change in the lives of individuals, in various work, family, and community settings. It's not by chance that Juliet Merrifield has played a strong research role with EFF as the EFF role maps are grounded in the concept of "literacy practices" as discussed by her and others, going back at least to Lytle and Wolfe in their overview of assessment in 1989 and also picked up in Fingeret's ERIC study of trends in 1992. This view also shares strong affinities with what Tom Sticht has referred to as functional context education, though there are some subtle (and highly significant) differences between Tom's version of contextual functional education and Juliet's concept of literacy practices at least as discussed in the 1997 monograph. With her work on EFF at least, I get the sense that Juliet has moved closer to the contextual functional position, though that may or may not be where she currently stands. The difference between the two perspectives of contextual literacy, briefly, is that the contextual functional perspective is linked more precisely to specific articulated goals, while the literacy as practices perspective as discussed in the Merrifield et al's book as well as in Fingeret and Drennon (1997) Life at the Margins, shows a much more indirect relationship between outcomes that emerge as a result of participating in programs and specific goals that students may or may not have had, going into the literacy program. I take something of this view in an essay titled "Motivation and the Adult New Reader: Case Profiles in a Deweyan Vein." (ABE Journal,Summer, 2001). Going back to Life at the Margins, one wonders what policy changed as a result of the initial report. One wonders also, if the strong governmental focus on workforce development, "objective" research, and the thrust toward quantitative, uniform, standardized assessment, delegitimizes the literacy as practices perspective in viewing literacy as an indirect variable or catalyst, from a policy-perspective as well as the invariable "response-bias" that accompanies ethnographic research. If so, perhaps such recent policy developments have resulted in a discounting or marginalization of the original policy study. From a certain paradigmatic slant, one could argue that from a strictly statistical perspective that the 12 case studies were not sufficiently representative in making a signifcant case about the impact of literacy. Be clear, though, that such a perspective would not be a reflection of "objective" science, but of one research tradition. These are some of the concerns of one practitioner, theorist-researcher. What gets legitimized as valid research by the policy sector and the circulatory impact of such legitimization on the field in reinforcing policy biases, is also a concern. The hope is that the National Academy of Sciences Board on Testing and Assessment will look at the issues somewhat widely and draw in part, on the research tradition that goes back to the late 80s and 90s as articulated by Susan Lytle, Hannah Fingeret, Allan Quigley, Juliet Merrifield, Elsa Auerbach, and others. This research tradition is based both on the assimilationist tendencies of the New Literacy Studies as well as the oppositional perspectives of critical pedagogy. These research traditions need to be engaged in tough dialogue with more normative interpretations that in theory, "an independent review of the state-of-the practice" of assessment accountability, would entail. Since it is inevitable, perhaps, that political ideology, will invariably influence the outcome (that assumes an ineradicable, though not deterministic assumption that power and knowledge are correlated), then such discussions need also to veer into political cultures that support or enhance certain research traditions. What also is contestable is the "levels metaphor" which grounds the current NRS. Some would argue that it is that metaphor itself, which needs to be changed. The broader issue (in my view) is not so much whether "valid" alternative assessment can be made congruent with the NRS (though that is the stated mandate of the Board on Testing and Assessment. Rather, the broader issue is whether an assessment system stemming from literacy as practices/contextual functional/emergent curriculum (Auerbach, Making Meaning) can be made congruent with a viable national reporting system. I hope you're seeing the difference there--the limitation being in defining the current National Reporting System through the research guise of "objectivity," standardization, and uniformity, when other research traditions, such as ethnography, for example, demonstrate a much more indirect relationship between adult literacy education and enhanced life efficacy. Coming to terms with these matters has less to do with what might be viewed as "legitimate" research, though it has everything to do with what might be viewed as legitimacy. At issue more than methodology and even specific research traditions, are values and political culture. Can an enlightened political culture be established in these United States of America that value the many indirect, but highly significant contributions that individuals draw from adult literacy education, which also has an impact on their public lives, whether in home, work, or community settings. A thorough exploration of some of these issues, particularly by mainstream research institutions, would, I argue, make an important contribution to the field and to practice. Whether there's a will for this, is anothermatter. George Demetrion Litercay Volunteers of Greater Hartford Gdemetruion@msn.com
This archive was generated by hypermail 2b30 : Fri Jan 18 2002 - 11:33:24 EST