Return-Path: <nifl-technology@literacy.nifl.gov> Received: from literacy (localhost [127.0.0.1]) by literacy.nifl.gov (8.10.2/8.10.2) with SMTP id iBFHqDU01603; Wed, 15 Dec 2004 12:52:13 -0500 (EST) Date: Wed, 15 Dec 2004 12:52:13 -0500 (EST) Message-Id: <3.0.6.32.20041215124818.00b988a8@mail.psnyc.com> Errors-To: listowner@literacy.nifl.gov Reply-To: nifl-technology@literacy.nifl.gov Originator: nifl-technology@literacy.nifl.gov Sender: nifl-technology@literacy.nifl.gov Precedence: bulk From: Mariann Fedele <mariannf@lacnyc.org> To: Multiple recipients of list <nifl-technology@literacy.nifl.gov> Subject: [NIFL-TECHNOLOGY:3378] FW: [NIFL-ASSESSMENT:820] Re: guidance in developing X-Listprocessor-Version: 6.0c -- ListProcessor by Anastasios Kotsikonas Content-Type: text/plain; charset="us-ascii" X-Mailer: QUALCOMM Windows Eudora Light Version 3.0.6 (32) Status: O Content-Length: 11354 Lines: 333 >From: "Marie Cora" <marie.cora@hotspurpartners.com> >To: "'Mariann Fedele'" <mariannf@lacnyc.org> >Subject: FW: [NIFL-ASSESSMENT:820] Re: guidance in developing job-related >Date: Wed, 15 Dec 2004 12:42:50 -0500 >X-MSMail-Priority: Normal >X-Mailer: Microsoft Outlook, Build 10.0.2627 >X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 >Importance: Normal >X-CanItPRO-Stream: lacnyc >X-Spam-Score: 0 () >X-Bayes-Prob: 0.0108 >X-Scanned-By: CanIt (www . canit . ca) > >And this one - start with Kevin's response below first, and then April's >reply. >Marie > >Ps: a quick reference for subscribing to the assessment discussion. Go >to: >http://literacy.kent.edu/Midwest/assessment/ >and click on Online Discussion either in the left toolbar or within the >text. > >-----Original Message----- >From: nifl-assessment@nifl.gov [mailto:nifl-assessment@nifl.gov] On >Behalf Of April L. Zenisky >Sent: Wednesday, December 15, 2004 9:03 AM >To: Multiple recipients of list >Subject: [NIFL-ASSESSMENT:820] Re: guidance in developing job-related > > >Hi Kevin, >So great to hear from you -- about a year to the day of our scoring >meeting in Amherst! > >Your questioning of the comparability of computer-based tests (CBTs) to >paper-and-pencil >counterparts is well-founded. I see a lot of research on this topic, >and often the results that are >out there are very dependent on a confluence of domain content and >population. Sometimes paper >scores are 'better' (i.e., higher), and sometimes the computer scores >are and there's not much rhyme >or reason to it, but one common thread seems to be that people _like_ >computerization more. Many >teachers and ABE students that I have spoken with recently have >expressed valid concerns about >fairness with computerized tests, but see how CBTs (and the practice >with computer skills that comes >along with them) have benefits for students --- for example, I've been >told that many employers >(Home Depot and others) now require most job applications to be >completed at computer workstations. > >On the issue of score comparability and the TOEFL, often scores from a >'new' computerized test are >not intended to be equivalent to the old paper-based test being >replaced. I am less familiar with >the specifics of the TOEFL testing program, but a similar example is the >American Institutes of >Certified Public Accountants CPA Exam. The old paper test was based on >test specifications from >some number of years ago, and the new computerized version (released >April 2004) was created in >response to a more recent job analysis that found that the information >entry-level accountants >needed had changed over the years, and that computers were an integral >part of how accountants were >doing their work. Thus, the logical choice for the CPAs was a >computerized assessment with funky >new formats (very performance-oriented with actual spreadsheets and tax >forms and the like embedded >in the test). I don't think that anyone at the AICPA, however, would >argue that the old paper and >new computerized CPA Exam scores are equivalent -- they measure somewhat >different things by design. > >I do know that the TOEFL folks did a lot of research into items types >and what kinds of skills >should be required by those test-takers, and the computer-adaptive test >that they came out with is >intended to reflect something a little different than the previous >version. > >Thanks also for bringing up the need for test administrator support -- >as procedures and practices >change from one mode of testing to another, _standardization_ between >test administrations (a >critical part of ensuring fairness for all test-takers) has to be >maintained. > >Cheers! >--april > > > >----- Original Message ----- >From: "Kevin O'Connor" <koconnor@framingham.k12.ma.us> >To: "Multiple recipients of list" <nifl-assessment@literacy.nifl.gov> >Sent: Tuesday, December 14, 2004 6:19 PM >Subject: [NIFL-ASSESSMENT:819] Re: guidance in developing job-related > > >> Hi April- >> >> This is a great question. So often we turn to computers to help us >solve our problems, taking for >> granted that using a computer relies on skills that are far from >universal. When the TOEFL >> switched to a computer-adaptive test I felt like I was the only one >who noticed that they were >> comparing apples and oranges- a lone voice crying in the wilderness. >> Suddenly students were required to write and essay on the computer >instead of on paper, and they >> could no longer look back at questions once they had passed. Working >in a second language is >> enough of a barrier to higher ed- forcing people to wait for computer >literacy on TOP of that was >> seen by many learners as yet another hurdle. I don't think we can >assume literacy. >> One thing I have realized in working with the BEST Plus is how much >Tech Support the TESTERS need- >> starting up a computer, opening a file, loading a CD, using a touch >pad, even plugging it in. >> These are teachers; they are people who have graduated college, people >with email accounts. We >> really need to be careful not to presume too much computer ability. >Just because it seems like >> second nature to me, using a computer relies on a whole new set of >reading, writing and cognitive >> skills compared to pencil and paper testing. >> >> Thanks for asking! >> >> Kevin O'Connor >> Assessment Specialist >> Framingham Adult ESL PLUS >> 508-626-4282 >> >> -----Original Message----- >> From: nifl-assessment@nifl.gov [mailto:nifl-assessment@nifl.gov]On >> Behalf Of April L. Zenisky >> Sent: Tuesday, December 14, 2004 2:04 PM >> To: Multiple recipients of list >> Subject: [NIFL-ASSESSMENT:815] Re: guidance in developing job-related >> >> >> >> Hi, all, >> To follow up with Marie's reposting, this was a question that was >especially interesting to me as >> my >> own research interests as a test developer involve alternate item >types (not multiple-choice) that >> are computer based. Specifically, I am interested in uses of >innovative or novel types that are >> highly engaging to the test-taker but still auto-scored. In many >cases, there might be one >> correct >> answer, but the questions might be written to allow for more than one >right (or wrong) answer. >> >> For example, the different item formats for use in computerized >testing allow you to incoporate >> different actions such as clicking, drag-and-drop, scrolling, and >typing. The drag-and-drop >> option >> is nice because people can rearrange onscreen items in different ways >given the frame of the >> question (for example, put things in some order (numerical, >alphabetical, chronological, etc.)) >> and >> it lets them visualize the groupings they are creating. >> >> So, here in Massachusetts my colleagues at the Center for Educational >Assessment and I are working >> with the Massachusetts DoE and practitioners in the state to develop >new, computerized assessments >> that are directly based on the Massachusetts curriculum. The MA >assessments will be different >> from >> the computerized BEST Plus in that the student willl enter answers >directly into the computer, >> which >> is different from the BEST Plus' approach (which works well for that >domain being tested -- a test >> administrator scores a student's spoken answers on a few dimensions >and enters those into the >> computer). >> >> One thing I'm interested in learning more about myself about is >people's experiences with adult >> learners and computerized testing. Some work I'm involved in now is >finding out about adult >> learners' familiarity with computerized response actions (clicking, >drag-and-drop, scrolling, and >> typing). Does anyone out there have any experience with or thoughts >about using computer >> technology >> for testing with adults that they'd be interested in sharing with the >list? More broadly, for >> practitioners, what do you find that your students can do with respect >to using a computer and >> what >> skills do you see them needing assistance with? What might you >consider to be the important >> navigational and other features of computerized >activities/assessments? >> >> I'm looking forward to hearing from you! >> --april >> >> >> >> *********************************************** >> April L. Zenisky, Ed.D. >> Director of Computer-Based Testing Initiatives >> Center for Educational Assessment >> University of Massachusetts Amherst >> Amherst, MA 01003 >> 413.577.2180 >> azenisky@educ.umass.edu >> >> >> >> >> >> >> >> >> >> ----- Original Message ----- >> From: "Marie Cora" <marie.cora@hotspurpartners.com> >> To: "Multiple recipients of list" <nifl-assessment@literacy.nifl.gov> >> Sent: Tuesday, December 14, 2004 12:50 PM >> Subject: [NIFL-ASSESSMENT:814] guidance in developing job-related >assessments for critical >> thinking >> >> >>> Hi everyone, >>> >>> I am re-posting Julie's question from October 14 because it is >relevant >>> to the discussion this week, and I don't think she ever received much >in >>> reply. Julie - have you made progress on this project to date? >Perhaps >>> April, and others who wish to, can comment on some of your questions. >>> marie >>> >>> >>> >>> I am working with large organization to develop assessments for >managers >>> in >>> training. They want the questions to focus on measuring whether or >not >>> a >>> management trainee can think critically and problem solve. The >questions >>> will focus >>> on IMPACT and INDICATIONS of daily company procedures. >>> >>> The types of questions need to assess if the trainee understands the >>> impact >>> of a particular procedure on the productivity, bottom-line, etc. for >the >>> >>> business. We also want to assess whether or not the trainees can >>> identify the >>> indicators that something is, or is not, going (or set up) according >to >>> procedures. >>> >>> For example, does the trainee understand the IMPACT if department >>> materials >>> are unorganized and/or dirty? That the mailbag has gone unopened? >Can >>> s/he >>> identify the INDICATORS that something is not running according to >>> procedure, is >>> out of place or that an employee is not performing their job >>> requirements? >>> >>> This assessment needs to written in standardized format (i.e., >multiple >>> choice, true false, some fill in the blank and a few short answers). >>> >>> I've created many assessments for basic skills, so have an >understanding >>> of >>> assessment development. However, I can see that testing intangible >>> problem >>> solving skills, will require a different approach. (By the way, the >>> precise >>> training modules have not been developed.) >>> >>> I am working with the company to identify the proficiencies, but >would >>> very >>> much appreciate samples of how to phrase the questions so that they >will >>> >>> accurately measure the proficiencies in the above mentioned format. >>> >>> Does anyone have samples or guidelines I might use? >>> >>> Thanks for your input. >>> >>> Julie Baker >>> >>> >>> >>> >>> >> >> > > > > Mariann Fedele Coordinator of Professional Development, Literacy Assistance Center Interim Moderator, NIFL Technology and Literacy Discussion List 32 Broadway 10th Floor New York, New York 10004 212-803-3325 mariannf@lacnyc.org www.lacnyc.org
This archive was generated by hypermail 2b30 : Thu Dec 23 2004 - 09:48:09 EST