Workplace colleagues,
I'm forwarding a post that came across the NIFL Assessment and Literacy Discussion List yesterday.  It has several back and forth posts included with the most recent post at the top.  The top one is the one I think may be of interest to some of you, but if you want to read the whole stream, begin with the bottom post.  They are discussing assessing literacy skills and the weaknesses of the NRS requirements in a workplace setting.
Donna
 
Donna Brian, Moderator
NIFL Workplace Discussion List
djgbrian at utk.edu

________________________________

From: assessment-bounces at nifl.gov on behalf of Lloyd David
Sent: Tue 10/23/2007 2:47 PM
To: 'The Assessment Discussion List'
Subject: [Assessment 998] Re: level movement



David,
I believe that one of the beautiful features of workplace education is the
need to include and somehow combine both academic evaluations and workplace
measures of change or progress. In the workplace education programs that CWL
offers there is always a TEAM comprised of representatives of the company
and the CWL program manager and if possible the instructor. One of the TEAMs
that we presently work with is composed of the HR manager, 2 supervisors
from different shifts, the plant manager, 2 production workers from
different shifts, the CWL program manager, and the CWL instructor. The TEAM
designed a notice about the program and informed the workforce about the
time and days for orientation and assessments. Initially we assessed 37
workers using CWL developed instruments in reading, writing, and speech. The
company had previously developed a math test. The TEAM selected 14 students
for 2 classes based on the results of the assessments. The TEAM had already
decided the skill levels desired for each class.  Later CWL will assess the
14 students using standardized test - BEST Plus and TABE. We will later
re-test the students using both CWL tests ,BEST Plus and the TABE to measure
academic changes. The members of the Team have discussed the goals of the
program and what they hope people will learn so as to improve productivity
on the job. The instructor is designing the curriculum to achieve these
goals as well as meeting the individual goals and needs of the students.
I did an evaluation study on a program CWL ran in a distribution center. The
study measured changes in attendance at work, error rates, and individual
benefits derived from participating in the classes. We interviewed each
student, the instructor, the supervisors, plant managers, and CFO (chief
financial officer). The study entitled "Value of Workplace Education" is on
a website www.creativeworkplacelearning.org The most significant changes
resulted from the increase in confidence as described by the students and
the supervisors. Errors decreased and attendance improved as measured
against a control group of similar workers who did  not attend the classes.
Lloyd 


Lloyd David, EdD.
Creative Workplace Learning
311 Washington Street
Brighton, MA 02135
Tel :  617-783-6360
FAX: 617-782-0136

-----Original Message-----
From: assessment-bounces at nifl.gov [mailto:assessment-bounces at nifl.gov] On
Behalf Of David J. Rosen
Sent: Saturday, October 20, 2007 10:04 AM
To: The Assessment Discussion List
Subject: [Assessment 994] Re: level movement

Hello Venu,

I don't think a longitudinal study, or a random sample of programs, would
meet the need that individual programs or students have for measuring
learner progress, or programs' interest in knowing learner outcomes or
impact on learners.  These need to be addressed in other ways.  But since
you have asked, here -- in broad brush strokes -- is what we need to do:

. Programs need formative and summative evaluation measures. A good
formative system should have as one of its goals no surprises for the
student and teacher when they see the results of a summative assessment.
. Standardized tests could be part of program evaluation if there is a high
degree of validity between what is taught and tested.
. Multiple measures are important.  In addition to standardized tests,
teacher evaluation, student self-evaluation, performance-based measures, and
other measures could be part of the mix.
. In adult education it is especially important to measure progress toward
and attainment of student goals and objectives. A superficial process of
asking students to check off their goals when they enroll is not sufficient.
Student goals need to regularly be re-visited, and broken down into
short-term objectives.  Students need an opportunity to regularly reflect on
their progress, perhaps with a 
teacher or tutor, or with other students. Their goals may change.  
indeed changing goals may be one measure of progress!

Does all of this sound like it would take more resources than most adult
education programs have? Yes.  High quality in education often requires more
resources.  If we believe in high quality, then we should advocate for it,
and for the resources needed to achieve it.

David J. Rosen
djrosen at comcast.net


---------------------------------------------


On Oct 19, 2007, at 6:55 PM, Venu Thelakkat wrote:


> David,

>

> An independent study of adult education students over a longer period

> of time sounds like a great way to evaluate the adult education system

> from a national or state perspective, even though, like you said,

> there are some methodological challenges.  It will certainly provide

> more valid and reliable data than we get from NRS.  However, it still

> leaves unresolved the issue of evaluating individual programs.  Like

> Diane said, states have been using NRS data to allocate funding and

> target technical assistance.  How would a longitudinal study meet this

> need?

>

> Venu Thelakkat



> Director of ASISTS/Data Analysis

 Literacy Assistance Center

> 32 Broadway, 10th floor

> New York, NY 10004

> (212) 803-3370

> venut at lacnyc.org

> www.lacnyc.org

>


---------------------------------------------------------------------------



> From: assessment-bounces at nifl.gov [mailto:assessment-

> bounces at nifl.gov] On Behalf Of David J. Rosen

> Sent: Thursday, October 18, 2007 8:38 PM

> To: The Assessment Discussion List

> Subject: [Assessment 984] Re: level movement



> Assessment Colleagues,

>

> On Oct 18, 2007, at 7:00 PM, Venu Thelakkat asked:

>

>> I am told that standardized assessments, especially those used in 

> adult literacy, have many problems.  But what is the alternative?  

>> Policy makers and funders (private or public) want accountability for

>> the money they disperse and there is very little else that has been

>> proposed.  Even in NRS, educational gain is the only measure that has

>> some validity as a program outcome.  The other measures such as

>> getting a job or entering post secondary education are very

>> unreliable given the wide variety of methods used to collect and

>> report the data.

>>

>

> Great question, Venu.

>

> One alternative that makes sense has been proposed, but I am told that

> Congress so far has been unwilling to accept it. Instead of reporting

> outcomes for every student in every federally-funded program, OVAE

> should each year pick a random sample of federally- funded programs

> and measure attainment of student goals over a multi-year period, that

> is, measure impact.  Some would argue that measuring impact is not

> possible, that adult education students are too hard to track over

> time, but the Longitudinal Study of Adult Literacy, now in its eighth

> year in Portland, Oregon, has shown that it is not impossible.

>

> Isn't impact more useful to policy makers than outcomes? Does anyone

> in Congress really care how many level gains students make?

> Some policy makers may care about how many program participants --

> especially over time -- have improved their employment status, can

> read to their children, can use  a computer, now have a diploma or

> GED, and have succeeded in their education beyond secondary level.  

> Of these goals, only the GED and high school diploma (for now at

> least) require a standardized test.

>

> David J. Rosen

> djrosen at comcast.net

>

> --------------------------------------------------------- <mailto:--------------------------------------------------------- at utk.edu> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.nifl.gov/pipermail/workplace/attachments/20071024/2d40185b/attachment.html