Return-Path: <nifl-assessment@literacy.nifl.gov> Received: from literacy (localhost [127.0.0.1]) by literacy.nifl.gov (8.10.2/8.10.2) with SMTP id i3P37xm16903; Sat, 24 Apr 2004 23:07:59 -0400 (EDT) Date: Sat, 24 Apr 2004 23:07:59 -0400 (EDT) Message-Id: <20040424.230150.6398.1.socrates555@juno.com> Errors-To: listowner@literacy.nifl.gov Reply-To: nifl-assessment@literacy.nifl.gov Originator: nifl-assessment@literacy.nifl.gov Sender: nifl-assessment@literacy.nifl.gov Precedence: bulk From: "George E. Demetrion" <socrates555@juno.com> To: Multiple recipients of list <nifl-assessment@literacy.nifl.gov> Subject: [NIFL-ASSESSMENT:530] The Gold Standard X-Listprocessor-Version: 6.0c -- ListProcessor by Anastasios Kotsikonas X-Mailer: Juno 1.49 Status: O Content-Length: 7495 Lines: 141 I had placed this on some of the other lists a while back, but in olieu of recent discussions on research traditions, perhaps this bearsb re-issuing here. George Demetrion ________________________________________________________ The Gold Standard In certain research circles randomized sampling, based on a "pure" experimental model of design is viewed as the gold standard of social science research. I would suggest if there is any gold standard it would be in the combination of the quality of question(s) asked, the designs that follows to best probe into the question(s), and the actual carrying out of any particular research project. A useful first question for advertisers might to use random sampling as a first cut to determine how many television viewers watched a particular program. Assuming there was a way to concretely identify a representative sub-sample of those viewers, product and interest surveys could be further given to get a better understanding of consumer purchasing decisions of that subset. That would still be a long way off between evaluating the impact on the advertising in relationship to the motivation and actual purchasing decisions of individuals who watched the program, though useful information en route would be gained. Needed from there would be in-depth focus group analysis in order to derive a qualitative assessment of the ways in which the advertisement influenced the viewers. From there, an imaginative leap into poetics would be needed in the advertising brain storming session in order to create new imagery that may better tap into core symbols of what would stimulate the relevant viewers to purchase the product or service in question. Thus even here there is no singular research methodology that reflects an objective gold standard of legitimate research, but a variety of methodologies that shift in importance based on the focus of the project throughout its various stages of evolution. Moreover, one does not necessarily need to start from the random sample of viewers. Rather, given the applied nature of the advertising research project, one might have started from an informal in-depth sample, or even from a hunch of the creator of the ad or the product manager of the good or service in question. The research project, then, would need to be further developed to gain the desired information of linking the ad to the motives and consumer preferences of the viewers. The point, after all, is to identify a good research question and then organize the research in order to obtain the information needed to best answer it. Again, given the applied nature of advertising research, a new ad might be created on what is viewed as sufficient research since ALL the data can never be collected (never mind analyzed) in any event. Choices and priorities have to be made all along the way, and in getting a handle on the nature of human motivation, complex qualitative issues need to be addressed as well as quantitative ones of measurement. Consequently, neither quantitative nor qualitative data are privileged per se. If such is the case with advertisement research, what about research on adult literacy? At least in some quarters the following hierarchy in adult literacy research quality is asserted: * Random sampling experimental design; the gold standard. * Quasi-experimental design; the silver standard * Correlational studies with statistical control; the bronze standard * Correlationational studies without statistical control; the copper stan= dard * Case study analysis (finally), the clay standard What stands out to me is a methodological fetishism as if the quality of research depends primarily on adhering as closely as possible to the defined gold standard. Thus, the research question, per se, is subordinated to this more fundamental methodological requirement. I find that curious, to say the least, except that I understand the politics of it, and thereby its logic. There are a lot of good research questions that could be posed. Given the marginality of resources supporting adult literacy, one good research question would focus on a longitudinal study of a select group of students who have made significant progress in their learning over, say, a three-year period. Progress is a complex phenomenon, which would need some type of definition. What would not be needed, however, is a uniform definition with the hope of isolating and controlling all the variables in order to precisely isolate one independent variable from another. That would not be needed as long as experimental design is not viewed as the gold standard of research. Some of the factors that would need to be addressed are the following: * First and foremost (perhaps), the definition of literacy: Whether the focus is primarily on reading and writing, or learning more broadly defined through the realm of print and verbal discourse. This would include learning both within and outside the program, and the impact of such learning on one's personal life and community impact. * The nature of the instructional program and amount of class time available. * First-hand observations and data analysis of student work over time. * The background, skills, and knowledge base of students, including reasons for their participation. * A detailed historical analysis of student learning, knowledge development and skill acquisition both within and outside the program within each of the years of the study. * A coherent composite descriptive analysis subject to further refinement= and alternative explanation. I've made the point in a previous posting that 100 carefully identified case studies would be substantially superior to 100,000 quantitative studies that did not significantly take the context of concrete learning scenarios into account. Allowing me some rhetorical license here, at the very least the empirical data would be quite different. Question: What would we as a field learn through a carefully developed analysis of 100 students representing different programs, entry-level points, ages, ethnic, racial, and gender backgrounds), focusing on those students, however variously defined, who have attained superior progress in a three year period? I emphasize students with superior progress in a highly marginalized field like ours in order to gain something of the outer boundaries of what could be accomplished through adult literacy if it were adequately resourced in the many ways that the field currently is not so supported. Such information, I argue, would likely yield valuable insight on best instructional and programmatic practices. That is at least what I am hypothesizing. Sub-questions: 1. What would be the variables that would need to be taken into account in developing a research project on this framework? 2. What methodologies would need to be drawn upon at what stages of the research project in order to be both rigorous and imaginative in design and to yield the optimal results? 3. If the result were a composite profile of the field that provided the field with a substantial roadmap on how best to organize programs and structure instruction, would the research be valuable even if it weren't shaped by an experimental design model? 4. Why is there such a fetish over methodology when there are so many issues of substance that needs to be addressed? 5. How can that be changed, and what are the costs of doing so and not doing so? George Demetrion
This archive was generated by hypermail 2b30 : Thu Dec 23 2004 - 09:46:15 EST