Return-Path: <nifl-assessment@literacy.nifl.gov> Received: from literacy (localhost [127.0.0.1]) by literacy.nifl.gov (8.10.2/8.10.2) with SMTP id i3CJCSm16916; Mon, 12 Apr 2004 15:12:29 -0400 (EDT) Date: Mon, 12 Apr 2004 15:12:29 -0400 (EDT) Message-Id: <Sea2-F46O925B76WE7P0001aaf2@hotmail.com> Errors-To: listowner@literacy.nifl.gov Reply-To: nifl-assessment@literacy.nifl.gov Originator: nifl-assessment@literacy.nifl.gov Sender: nifl-assessment@literacy.nifl.gov Precedence: bulk From: "Eileen Eckert" <eileeneckert@hotmail.com> To: Multiple recipients of list <nifl-assessment@literacy.nifl.gov> Subject: [NIFL-ASSESSMENT:502] Re: why "valid and reliable"? X-Listprocessor-Version: 6.0c -- ListProcessor by Anastasios Kotsikonas Content-Type: text/plain; format=flowed Status: O Content-Length: 6194 Lines: 117 Hi Karen, I don't know if you did come in halfway through, but I for one welcome your entry into the discussion. Now I'm carrying on at least 3 assessment conversations in my head and can blame the number of different points I'm trying to keep straight if any one of my replies doesn't make sense ; ) I wanted to look at the assumptions behind "validity and reliability" as criteria for judging the quality of assessment because I think that we should be using "trustworthiness" with its components of credibility, dependability, transferability, and generalizability instead. Just as naturalistic research is an alternative to positivistic research, naturalistic assessment should be an alternative to positivist assessment. But in order for it to become one, we have to recognize that validity and reliability fall within a positivist paradigm, and that there are alternatives to that paradigm. You said, "You seem to be assuming that 'skills cannot be detached from their context'." Not quite, but I did say that transfer of skills is a difficult issue, and that just because someone can use a skill in one context doesn't <necessarily> mean they can use it in another. It doesn't necessarily mean they can't either, but if you want to know if someone can apply knowledge, understanding, or skill broadly across contexts, then I think you need to give them the opportunity to do so and assess and document the extent to which they can, not give them a single item on a test and assume that a correct response not only represents the skill, but also the ability to transfer it. On the use of the word "skill", I'm probably using it too loosely. The use of the word "skill" may bother you the way the use of the term "competency" and the narrowness of what it represents--a kind of checklist in which the sum of the parts add up to a whole of expertise, and which excludes much tacit, metacognitive, and strategic knowledge--bothers me. But that's another discussion thread, I think. Eileen From: HthKar@aol.com Reply-To: nifl-assessment@nifl.gov To: Multiple recipients of list <nifl-assessment@literacy.nifl.gov> Subject: [NIFL-ASSESSMENT:499] Re: why "valid and reliable"? Date: Mon, 12 Apr 2004 09:46:47 -0400 (EDT) EIleen Pardon me for coming into a debate half way through: You are quite right to point out that the concepts of reliability and validity passed from psychometric testing, where they were I think developed (Binet etc etc). into social science and that they can be 'critiqued'. Some theorists seem to me to have 'disappeared up their own research methods' some decades ago. Personally I generally prefer not to waste my time reading anything written by someone who thinks that the real word does not exist: on their own arguments why should I bother? The trick is to read the last chaper first; they often start off acting normal and get weird as they go along. I would also prefer not to pay taxes to support them in the luxury to which they have become accustomed, but there you are.. I sometimes wonder whether some people would maintain various 'relative' or nihilistic positions if, say, the vocational person who washed their windows ironically put them through instead claiming he was just being 'para...lytic??' Worst of all, some social scientists seem to have decided to be poetical. I am thinking that we shall have to coin a new term, not doggerel but proserel perhaps. I suspect that this could be a very circular argument based on a very 'situated' definition of some thing or other. For example, there is a chap on the boulevard outside mending his car. The neighborhood has of course requested him to stop letting the tone of the place down but in vain. However, he does not appear to be doing it because he cannot detach his car-mending skills from the context of the boulevard. Though when asked 'whether he could do that somewhere else' he replied using delightfully colourful vernacular language in the negative, this seems to have been because of the ambiguity of the modal in question. Not only that, but last year he was applying his car-mending skills to a completely different model of car. There is a chap down the road who sits on his step sewing in sunny weather. He says he learned how to do this in (WEST?) Africa somewhere as an apprentice. Another point in what you wrote that I picked up on is this use of the word 'skills' which I think tends to obscure a great variety in what people know, understand and can do. The very use of the word 'skill' to describe the ability to read and to write obscures the subject and also leads to unhelpful conceptualisations. One example of this is the insistence that filling forms in is a 'transferable skill' I recently attended a training session in Barsetshire run by an awarding body in which the handout stated that filling in forms is a 'fundamental transfereble skill'. A person who directed courses on which people were trained to train other people to train people to read and write bu who had no teaching competence themselves once insisted to me that this was the case, and specifically that once you knew how to fill in one form you could fill them all in. I concluded that this firm belief probably resulted from having seen 'complete a simple' form listed as a 'competen! cy' on various accreditation frameworks: people tend to equate 'competencies' with 'skills'..... Of course people can transfer what they have learned to other situations: if there were no transfer then there could be no learning at all. Nothing could 'transfer' to any other situation. This is fundamental. On the other hand, if I accepted what you said about its being impossible to detach skills from their context, then I would have to say to you how could one start to critique the concepts of validity and reliability unless one had previously 'learned' to do the same thing in the same context? Cheers Karen _________________________________________________________________ Check out MSN PC Safety & Security to help ensure your PC is protected and safe. http://specials.msn.com/msn/security.asp
This archive was generated by hypermail 2b30 : Thu Dec 23 2004 - 09:46:14 EST