Workplace discussion list participants,

I am cross-posting a discussion from the NIFL Assessment Discussion List 
because it is a good example of how the Workplace and Assessment interests 
intersect.  If any of you want to join this discussion from the workplace 
standpoint, you can just answer on the workplace list and I will crosspost 
your response to the Assessment list.  (Alternatively, you could join the 
Assessment Discussion list and post directly.)

Donna JG Brian
Moderator, NIFL Workplace Literacy Discussion List, and
Coordinator/Developer LINCS Workforce Education Special Collection at 
http://worklink.coe.utk.edu/
Center for Literacy Studies at The University of Tennessee
600 Henley Street, Suite 312
Knoxville, TN 37996-4135

865-974-3420 (desk phone)        FAX 865-974-3857
djgbrian at utk.edu



>Date: Wed, 15 Dec 2004 09:04:46 -0500 (EST)

>From: "April L. Zenisky" <azenisky at educ.umass.edu>

>Subject: [NIFL-ASSESSMENT:820] Re: guidance in developing job-related

>Sender: nifl-assessment at nifl.gov

>To: Multiple recipients of list <nifl-assessment at literacy.nifl.gov>

>

>

>Hi Kevin,

>So great to hear from you -- about a year to the day of our scoring 

>meeting in Amherst!

>

>Your questioning of the comparability of computer-based tests (CBTs) to 

>paper-and-pencil counterparts is well-founded.  I see a lot of research on 

>this topic, and often the results that are out there are very dependent on 

>a confluence of domain content and population.  Sometimes paper scores are 

>'better' (i.e., higher), and sometimes the computer scores are and there's 

>not much rhyme or reason to it, but one common thread seems to be that 

>people _like_ computerization more.  Many teachers and ABE students that I 

>have spoken with recently have expressed valid concerns about fairness 

>with computerized tests, but see how CBTs (and the practice with computer 

>skills that comes along with them) have benefits for students --- for 

>example, I've been told that many employers (Home Depot and others) now 

>require most job applications to be completed at computer workstations.

>

>On the issue of score comparability and the TOEFL, often scores from a 

>'new' computerized test are not intended to be equivalent to the old 

>paper-based test being replaced.  I am less familiar with the specifics of 

>the TOEFL testing program, but a similar example is the American 

>Institutes of Certified Public Accountants CPA Exam.  The old paper test 

>was based on test specifications from some number of years ago, and the 

>new computerized version (released April 2004) was created in response to 

>a more recent job analysis that found that the information entry-level 

>accountants needed had changed over the years, and that computers were an 

>integral part of how accountants were doing their work.  Thus, the logical 

>choice for the CPAs was a computerized assessment with funky new formats 

>(very performance-oriented with actual spreadsheets and tax forms and the 

>like embedded in the test).  I don't think that anyone at the AICPA, 

>however, would argue that the old paper and new computerized CPA Exam 

>scores are equivalent -- they measure somewhat different things by design.

>

>I do know that the TOEFL folks did a lot of research into items types and 

>what kinds of skills should be required by those test-takers, and the 

>computer-adaptive test that they came out with is intended to reflect 

>something a little different than the previous version.

>

>Thanks also for bringing up the need for test administrator support -- as 

>procedures and practices change from one mode of testing to another, 

>_standardization_ between test administrations (a critical part of 

>ensuring fairness for all test-takers) has to be maintained.

>

>Cheers!

>--april

>

>

>

>----- Original Message ----- From: "Kevin O'Connor" 

><koconnor at framingham.k12.ma.us>

>To: "Multiple recipients of list" <nifl-assessment at literacy.nifl.gov>

>Sent: Tuesday, December 14, 2004 6:19 PM

>Subject: [NIFL-ASSESSMENT:819] Re: guidance in developing job-related

>

>

>>Hi April-

>>

>>This is a great question.  So often we turn to computers to help us solve 

>>our problems, taking for granted that using a computer relies on skills 

>>that are far from universal.  When the TOEFL switched to a 

>>computer-adaptive test I felt like I was the only one who noticed that 

>>they were comparing apples and oranges- a lone voice crying in the wilderness.

>>Suddenly students were required to write and essay on the computer 

>>instead of on paper, and they could no longer look back at questions once 

>>they had passed.  Working in a second language is enough of a barrier to 

>>higher ed- forcing people to wait for computer literacy on TOP of that 

>>was seen by many learners as yet another hurdle. I don't think we can 

>>assume literacy.

>>One thing I have realized in working with the BEST Plus is how much Tech 

>>Support the TESTERS need- starting up a computer, opening a file, loading 

>>a CD, using a touch pad, even plugging it in. These are teachers; they 

>>are people who have graduated college, people with email accounts.  We 

>>really need to be careful not to presume too much computer ability.  Just 

>>because it seems like second nature to me, using a computer relies on a 

>>whole new set of reading, writing and cognitive skills compared to pencil 

>>and paper testing.

>>

>>Thanks for asking!

>>

>>Kevin O'Connor

>>Assessment Specialist

>>Framingham Adult ESL PLUS

>>508-626-4282

>>

>>-----Original Message-----

>>From: nifl-assessment at nifl.gov [mailto:nifl-assessment at nifl.gov]On

>>Behalf Of April L. Zenisky

>>Sent: Tuesday, December 14, 2004 2:04 PM

>>To: Multiple recipients of list

>>Subject: [NIFL-ASSESSMENT:815] Re: guidance in developing job-related

>>

>>

>>

>>Hi, all,

>>To follow up with Marie's reposting, this was a question that was 

>>especially interesting to me as my

>>own research interests as a test developer involve alternate item types 

>>(not multiple-choice) that

>>are computer based.  Specifically, I am interested in uses of innovative 

>>or novel types that are

>>highly engaging to the test-taker but still auto-scored.  In many cases, 

>>there might be one correct

>>answer, but the questions might be written to allow for more than one 

>>right (or wrong) answer.

>>

>>For example, the different item formats for use in computerized testing 

>>allow you to incoporate

>>different actions such as clicking, drag-and-drop, scrolling, and 

>>typing.  The drag-and-drop option

>>is nice because people can rearrange onscreen items in different ways 

>>given the frame of the

>>question (for example, put things in some order (numerical, alphabetical, 

>>chronological, etc.)) and

>>it lets them visualize the groupings they are creating.

>>

>>So, here in Massachusetts my colleagues at the Center for Educational 

>>Assessment and I are working

>>with the Massachusetts DoE and practitioners in the state to develop new, 

>>computerized assessments

>>that are directly based on the Massachusetts curriculum.  The MA 

>>assessments will be different from

>>the computerized BEST Plus in that the student willl enter answers 

>>directly into the computer, which

>>is different from the BEST Plus' approach (which works well for that 

>>domain being tested -- a test

>>administrator scores a student's spoken answers on a few dimensions and 

>>enters those into the

>>computer).

>>

>>One thing I'm interested in learning more about myself about is people's 

>>experiences with adult

>>learners and computerized testing.  Some work I'm involved in now is 

>>finding out about adult

>>learners' familiarity with computerized response actions (clicking, 

>>drag-and-drop, scrolling, and

>>typing).  Does anyone out there have any experience with or thoughts 

>>about using computer technology

>>for testing with adults that they'd be interested in sharing with the 

>>list?  More broadly, for

>>practitioners, what do you find that your students can do with respect to 

>>using a computer and what

>>skills do you see them needing assistance with?  What might you consider 

>>to be the important

>>navigational and other features of computerized activities/assessments?

>>

>>I'm looking forward to hearing from you!

>>--april

>>

>>

>>

>>***********************************************

>>April L. Zenisky, Ed.D.

>>Director of Computer-Based Testing Initiatives

>>Center for Educational Assessment

>>University of Massachusetts Amherst

>>Amherst, MA 01003

>>413.577.2180

>>azenisky at educ.umass.edu

>>

>>

>>

>>

>>

>>

>>

>>

>>

>>----- Original Message ----- From: "Marie Cora" 

>><marie.cora at hotspurpartners.com>

>>To: "Multiple recipients of list" <nifl-assessment at literacy.nifl.gov>

>>Sent: Tuesday, December 14, 2004 12:50 PM

>>Subject: [NIFL-ASSESSMENT:814] guidance in developing job-related 

>>assessments for critical thinking

>>

>>

>>>Hi everyone,

>>>

>>>I am re-posting Julie's question from October 14 because it is relevant

>>>to the discussion this week, and I don't think she ever received much in

>>>reply.  Julie - have you made progress on this project to date?  Perhaps

>>>April, and others who wish to, can comment on some of your questions.

>>>marie

>>>

>>>

>>>

>>>I am working with large organization to develop assessments for managers

>>>in

>>>training.  They want the questions to focus on measuring whether or not

>>>a

>>>management trainee can think critically and problem solve. The questions

>>>will focus

>>>on IMPACT and INDICATIONS of daily company procedures.

>>>

>>>The types of questions need to assess if the trainee understands the

>>>impact

>>>of a particular procedure on the productivity, bottom-line, etc. for the

>>>

>>>business.  We also want to assess whether or not the trainees can

>>>identify the

>>>indicators that something is, or is not, going (or set up) according to

>>>procedures.

>>>

>>>For example, does the trainee understand the IMPACT if department

>>>materials

>>>are unorganized and/or dirty?  That the mailbag has gone unopened?  Can

>>>s/he

>>>identify the INDICATORS that something is not running according to

>>>procedure, is

>>>out of place or that an employee is not performing their job

>>>requirements?

>>>

>>>This assessment needs to written in standardized format (i.e., multiple

>>>choice, true false, some fill in the blank and a few short answers).

>>>

>>>I've created many assessments for basic skills, so have an understanding

>>>of

>>>assessment development. However, I can see that testing intangible

>>>problem

>>>solving skills, will require a different approach. (By the way, the

>>>precise

>>>training modules have not been developed.)

>>>

>>>I am working with the company to identify the proficiencies, but would

>>>very

>>>much appreciate samples of how to phrase the questions so that they will

>>>

>>>accurately measure the proficiencies in the above mentioned format.

>>>

>>>Does anyone have samples or guidelines I might use?

>>>

>>>Thanks for your input.

>>>

>>>Julie Baker

>>>

>>>

>>>

>>>

>>