Return-Path: <nifl-assessment@literacy.nifl.gov> Received: from literacy (localhost [127.0.0.1]) by literacy.nifl.gov (8.10.2/8.10.2) with SMTP id iBFJ1jU03138; Wed, 15 Dec 2004 14:01:45 -0500 (EST) Date: Wed, 15 Dec 2004 14:01:45 -0500 (EST) Message-Id: <5.1.0.14.2.20041215135656.02869e30@pop.utk.edu> Errors-To: listowner@literacy.nifl.gov Reply-To: nifl-assessment@literacy.nifl.gov Originator: nifl-assessment@literacy.nifl.gov Sender: nifl-assessment@literacy.nifl.gov Precedence: bulk From: Donna Brian <djgbrian@utk.edu> To: Multiple recipients of list <nifl-assessment@literacy.nifl.gov> Subject: [NIFL-ASSESSMENT:824] Re: guidance in developing X-Listprocessor-Version: 6.0c -- ListProcessor by Anastasios Kotsikonas Content-Type: text/plain; charset="us-ascii"; format=flowed X-Mailer: QUALCOMM Windows Eudora Version 5.1 Status: O Content-Length: 11444 Lines: 324 I cross-posted to the Workplace discussion list also. If there are replies that speak to assessment issues, I'll cross-post them on this list. Donna Brian Workplace Discussion List Moderator djgbrian@utk.edu At 12/15/2004 -050012:49 PM, you wrote: >Hi folks, > >Great discussion! Thanks! I just wanted to let you know that I am >cross-posting some of the messages to the Technology & Literacy List >(http://www.nifl.gov/lincs/discussions/nifl-technology/technology_litera >cy.html). I was thinking that people over there might have some >comments and suggestions about the more general questions around the ABE >student and technology (comfort levels, challenges, successes, what >works, etc.). So we may get some replies from that group, I hope, that >would be great. > >Thanks, >marie cora >Moderator, NIFL Assessment Discussion List, and >Coordinator/Developer LINCS Assessment Special Collection at >http://literacy.kent.edu/Midwest/assessment/ > > >marie.cora@hotspurpartners.com > >-----Original Message----- >From: nifl-assessment@nifl.gov [mailto:nifl-assessment@nifl.gov] On >Behalf Of April L. Zenisky >Sent: Wednesday, December 15, 2004 9:03 AM >To: Multiple recipients of list >Subject: [NIFL-ASSESSMENT:820] Re: guidance in developing job-related > > >Hi Kevin, >So great to hear from you -- about a year to the day of our scoring >meeting in Amherst! > >Your questioning of the comparability of computer-based tests (CBTs) to >paper-and-pencil >counterparts is well-founded. I see a lot of research on this topic, >and often the results that are >out there are very dependent on a confluence of domain content and >population. Sometimes paper >scores are 'better' (i.e., higher), and sometimes the computer scores >are and there's not much rhyme >or reason to it, but one common thread seems to be that people _like_ >computerization more. Many >teachers and ABE students that I have spoken with recently have >expressed valid concerns about >fairness with computerized tests, but see how CBTs (and the practice >with computer skills that comes >along with them) have benefits for students --- for example, I've been >told that many employers >(Home Depot and others) now require most job applications to be >completed at computer workstations. > >On the issue of score comparability and the TOEFL, often scores from a >'new' computerized test are >not intended to be equivalent to the old paper-based test being >replaced. I am less familiar with >the specifics of the TOEFL testing program, but a similar example is the >American Institutes of >Certified Public Accountants CPA Exam. The old paper test was based on >test specifications from >some number of years ago, and the new computerized version (released >April 2004) was created in >response to a more recent job analysis that found that the information >entry-level accountants >needed had changed over the years, and that computers were an integral >part of how accountants were >doing their work. Thus, the logical choice for the CPAs was a >computerized assessment with funky >new formats (very performance-oriented with actual spreadsheets and tax >forms and the like embedded >in the test). I don't think that anyone at the AICPA, however, would >argue that the old paper and >new computerized CPA Exam scores are equivalent -- they measure somewhat >different things by design. > >I do know that the TOEFL folks did a lot of research into items types >and what kinds of skills >should be required by those test-takers, and the computer-adaptive test >that they came out with is >intended to reflect something a little different than the previous >version. > >Thanks also for bringing up the need for test administrator support -- >as procedures and practices >change from one mode of testing to another, _standardization_ between >test administrations (a >critical part of ensuring fairness for all test-takers) has to be >maintained. > >Cheers! >--april > > > >----- Original Message ----- >From: "Kevin O'Connor" <koconnor@framingham.k12.ma.us> >To: "Multiple recipients of list" <nifl-assessment@literacy.nifl.gov> >Sent: Tuesday, December 14, 2004 6:19 PM >Subject: [NIFL-ASSESSMENT:819] Re: guidance in developing job-related > > > > Hi April- > > > > This is a great question. So often we turn to computers to help us >solve our problems, taking for > > granted that using a computer relies on skills that are far from >universal. When the TOEFL > > switched to a computer-adaptive test I felt like I was the only one >who noticed that they were > > comparing apples and oranges- a lone voice crying in the wilderness. > > Suddenly students were required to write and essay on the computer >instead of on paper, and they > > could no longer look back at questions once they had passed. Working >in a second language is > > enough of a barrier to higher ed- forcing people to wait for computer >literacy on TOP of that was > > seen by many learners as yet another hurdle. I don't think we can >assume literacy. > > One thing I have realized in working with the BEST Plus is how much >Tech Support the TESTERS need- > > starting up a computer, opening a file, loading a CD, using a touch >pad, even plugging it in. > > These are teachers; they are people who have graduated college, people >with email accounts. We > > really need to be careful not to presume too much computer ability. >Just because it seems like > > second nature to me, using a computer relies on a whole new set of >reading, writing and cognitive > > skills compared to pencil and paper testing. > > > > Thanks for asking! > > > > Kevin O'Connor > > Assessment Specialist > > Framingham Adult ESL PLUS > > 508-626-4282 > > > > -----Original Message----- > > From: nifl-assessment@nifl.gov [mailto:nifl-assessment@nifl.gov]On > > Behalf Of April L. Zenisky > > Sent: Tuesday, December 14, 2004 2:04 PM > > To: Multiple recipients of list > > Subject: [NIFL-ASSESSMENT:815] Re: guidance in developing job-related > > > > > > > > Hi, all, > > To follow up with Marie's reposting, this was a question that was >especially interesting to me as > > my > > own research interests as a test developer involve alternate item >types (not multiple-choice) that > > are computer based. Specifically, I am interested in uses of >innovative or novel types that are > > highly engaging to the test-taker but still auto-scored. In many >cases, there might be one > > correct > > answer, but the questions might be written to allow for more than one >right (or wrong) answer. > > > > For example, the different item formats for use in computerized >testing allow you to incoporate > > different actions such as clicking, drag-and-drop, scrolling, and >typing. The drag-and-drop > > option > > is nice because people can rearrange onscreen items in different ways >given the frame of the > > question (for example, put things in some order (numerical, >alphabetical, chronological, etc.)) > > and > > it lets them visualize the groupings they are creating. > > > > So, here in Massachusetts my colleagues at the Center for Educational >Assessment and I are working > > with the Massachusetts DoE and practitioners in the state to develop >new, computerized assessments > > that are directly based on the Massachusetts curriculum. The MA >assessments will be different > > from > > the computerized BEST Plus in that the student willl enter answers >directly into the computer, > > which > > is different from the BEST Plus' approach (which works well for that >domain being tested -- a test > > administrator scores a student's spoken answers on a few dimensions >and enters those into the > > computer). > > > > One thing I'm interested in learning more about myself about is >people's experiences with adult > > learners and computerized testing. Some work I'm involved in now is >finding out about adult > > learners' familiarity with computerized response actions (clicking, >drag-and-drop, scrolling, and > > typing). Does anyone out there have any experience with or thoughts >about using computer > > technology > > for testing with adults that they'd be interested in sharing with the >list? More broadly, for > > practitioners, what do you find that your students can do with respect >to using a computer and > > what > > skills do you see them needing assistance with? What might you >consider to be the important > > navigational and other features of computerized >activities/assessments? > > > > I'm looking forward to hearing from you! > > --april > > > > > > > > *********************************************** > > April L. Zenisky, Ed.D. > > Director of Computer-Based Testing Initiatives > > Center for Educational Assessment > > University of Massachusetts Amherst > > Amherst, MA 01003 > > 413.577.2180 > > azenisky@educ.umass.edu > > > > > > > > > > > > > > > > > > > > ----- Original Message ----- > > From: "Marie Cora" <marie.cora@hotspurpartners.com> > > To: "Multiple recipients of list" <nifl-assessment@literacy.nifl.gov> > > Sent: Tuesday, December 14, 2004 12:50 PM > > Subject: [NIFL-ASSESSMENT:814] guidance in developing job-related >assessments for critical > > thinking > > > > > >> Hi everyone, > >> > >> I am re-posting Julie's question from October 14 because it is >relevant > >> to the discussion this week, and I don't think she ever received much >in > >> reply. Julie - have you made progress on this project to date? >Perhaps > >> April, and others who wish to, can comment on some of your questions. > >> marie > >> > >> > >> > >> I am working with large organization to develop assessments for >managers > >> in > >> training. They want the questions to focus on measuring whether or >not > >> a > >> management trainee can think critically and problem solve. The >questions > >> will focus > >> on IMPACT and INDICATIONS of daily company procedures. > >> > >> The types of questions need to assess if the trainee understands the > >> impact > >> of a particular procedure on the productivity, bottom-line, etc. for >the > >> > >> business. We also want to assess whether or not the trainees can > >> identify the > >> indicators that something is, or is not, going (or set up) according >to > >> procedures. > >> > >> For example, does the trainee understand the IMPACT if department > >> materials > >> are unorganized and/or dirty? That the mailbag has gone unopened? >Can > >> s/he > >> identify the INDICATORS that something is not running according to > >> procedure, is > >> out of place or that an employee is not performing their job > >> requirements? > >> > >> This assessment needs to written in standardized format (i.e., >multiple > >> choice, true false, some fill in the blank and a few short answers). > >> > >> I've created many assessments for basic skills, so have an >understanding > >> of > >> assessment development. However, I can see that testing intangible > >> problem > >> solving skills, will require a different approach. (By the way, the > >> precise > >> training modules have not been developed.) > >> > >> I am working with the company to identify the proficiencies, but >would > >> very > >> much appreciate samples of how to phrase the questions so that they >will > >> > >> accurately measure the proficiencies in the above mentioned format. > >> > >> Does anyone have samples or guidelines I might use? > >> > >> Thanks for your input. > >> > >> Julie Baker > >> > >> > >> > >> > >> > > > >
This archive was generated by hypermail 2b30 : Thu Dec 23 2004 - 09:46:27 EST