[NIFL-ASSESSMENT:544] Cross-post from NIFL-PLI: Using Data for Program Improvement

mariecora@hotmail.com

Date: Wed, 12 May 2004 11:20:56 -0400 (EDT)
Message-Id: <BAY17-F10HgS2llAwN00003f600@hotmail.com>
Errors-To: listowner@literacy.nifl.gov
Reply-To: nifl-assessment@literacy.nifl.gov
Originator: nifl-assessment@literacy.nifl.gov
Sender: nifl-assessment@literacy.nifl.gov
Precedence: bulk
From: "Marie Cora" <mariecora@hotmail.com>
To: Multiple recipients of list <nifl-assessment@literacy.nifl.gov>
Subject: [NIFL-ASSESSMENT:544] Cross-post from NIFL-PLI:  Using Data for Program Improvement
Content-Type: text/plain; format=flowed
Status: O
Content-Length: 16445
Lines: 332

Hi everyone,

Warning:  this is a lengthy post.  But for those who are interested in this 
topic, in particular in regards to:

-responding to the NRS
-if/how analyzing program data is done in your program or even seen as 
useful

then this is a good thing for you to read.  There are a couple of comments 
by program folks in response to an overview by Larrry Condelli of his work 
with the NRS in which he poses a series of questions about using data for 
program improvement at the end.

For a couple of articles about using data, go to:

http://literacy.kent.edu/Midwest/assessment/adventures_vol13.html
or click on Adventures in Assessment at the NIFL Assessment Collection and 
look for Volume 13.

thanks,
marie cora
NIFL Assessment List Moderator

>From: Sandy Strunk <sandy_strunk@iu13.org>
>Reply-To: Sandy Strunk <sandy_strunk@iu13.org>,   Program Leadership and 
>Improvement Discussion List<nifl-pli@nifl.gov>
>To: Leadership and Improvement Dis <nifl-pli@nifl.gov>
>Subject: RE: [NIFL-PLI] Re: NIFL-PLI Digest, Vol 8, Issue 1
>Date: 11 May 2004 11:10:45 -0700
>
>          Reply to:   RE: [NIFL-PLI] Re: NIFL-PLI Digest, Vol 8, Issue 1
>Sonia makes some interesting points that I'd like to turn back to the group 
>for discussion. I've been doing this work for almost twenty years now, but 
>it's hard for me to remember the days before program accountability was a 
>primary focus for those of us who are leaders in our field. None-the-less, 
>there was such a time and it wasn't as if we all sat around waiting for 
>something to do or for someone to invent an adult education accountability 
>system.
>Before the advent of the NRS, which data, if any, WERE programs actually 
>collecting and analyzing data as a basis for program improvement? We often 
>talk about data for decision-making as if it were a no-brainer, but in my 
>experience, not every program buys into ongoing data collection and 
>analysis as a basis for continuous improvement.
>
>Last night, I was reading a dissertation by Priscilla Sissem, one of my PA 
>colleagues, titled, "Leaders of Adult Basic and Literacy Education Programs 
>in Pennsylvania: A Typology of Leadership Styles and Organizational Issues 
>in Context."  Her premise is that leaders in our field can be broken into 
>three typologies:  Founders, Mainstreamers, and Prospectors -- with each 
>type approaching this work somewhat differently. She notes that whereas 
>some program leaders believe that increased accountability has set a higher 
>standard for our field, has brought us credibility and a new 
>professionalism, others suggest that increased accountability has 
>undermined one of our most deeply held commitments - the commitment to 
>learners as individuals and to each learner's need for individualized 
>instruction. This tension is fascinating and certainly underscores the 
>diversity of the leadership in our field.
>
>Larry's questions about how we use data for program improvement are based 
>on the assumption that program leaders actually find value in collecting 
>and analyzing data as a basis for decision-making. Clearly, some do and 
>others do not.  I'm wondering if any discussion related to using the NRS 
>for program improvement might need to start with an exploration of state 
>and federal accountability systems in general and the impact they have had, 
>and continue to have, on our field.
>
>Any thoughts?
>
>
>Sandy Strunk
>Lancaster Lebanon Intermediate Unit 13
>Adult Basic and Family Literacy Education
>1110 Enterprise Road
>East Petersburg, PA 17520
>(717) 519-1006
>
>Sonia Socha wrote:
> >This overview from Larry was excellent information for a provider in the 
> >field to pass on to staff who are trying to implement the NRS.
> >
> >The questions at the end will yake along time to answer and I prefer not 
>to >write.  Would much rather have a personal conversation with Mr. 
>Condelli.
> >
> >In many ways, our program was way ahead of NRS intitially with collecting 
> >data and making program improvements based on our collections--but what 
>we collect >and what we look for is slightly different than what the feds 
>are looking for.  >And , of course, there are those items that are the 
>same--but we anaylze them >differently.
> >
> >Do not have the time to answer these questions now--too much on the plate 
> >in the month of May--especially completing grants for adult literacy 
>funding!  We >are also preparing for a visit from USDE in June--I know our 
>staff will have much >to say about data collection during that monitoring 
>visit.
> >
> >Happy to talk sometime over the summer--I am sure our program will have 
> >much to contribute to the questions outlined below.
> >
> >Thank you for the opportunity.   >
> >>>> nifl-pli-request@nifl.gov 05/10/04 12:00PM >>>
> >Send NIFL-PLI mailing list submissions to
> >	nifl-pli@literacy.nifl.gov >
> >To subscribe or unsubscribe via the World Wide Web, visit
> >	http://www.nifl.gov/mailman/listinfo/nifl-pli >or, via email, send a 
>message with subject or body 'help' to
> >	nifl-pli-request@literacy.nifl.gov >
> >You can reach the person managing the list at
> >	nifl-pli-owner@literacy.nifl.gov >
> >When replying, please edit your Subject line so it is more specific
> >than "Re: Contents of NIFL-PLI digest..."
> >
> >
> >Today's Topics:
> >
> >   1. Using NRS Data for Program Improvement (Sandy Strunk)
> >
> >
> >----------------------------------------------------------------------
> >
> >Message: 1
> >Date: 10 May 2004 07:59:03 -0700
> >From: Sandy Strunk <sandy_strunk@iu13.org>
> >Subject: [NIFL-PLI] Using NRS Data for Program Improvement
> >To: "NIFL -PLI List" <nifl-pli@literacy.nifl.gov>
> >Message-ID: <20040510115616.D0950470023@relay.iu13.org>
> >Content-Type: text/plain; charset="iso-8859-1"
> >
> >The following posting is from Larry Condelli, Managing Director in the 
> >Education and Human Development Division of the American Institues for 
>Research. >Larry leads the adult education team and is project director for 
>the National >Reporting System. He has been involved in projects on ESL 
>research, the professional >development of adult educators and the 
>development of content standards for adult >education. He has worked in 
>adult education since 1990.
> >
> >At the end of his posting, Larry poses several questions he hopes you'll 
> >respond to. I, too, hope you will take this opportunity to discuss the 
>National >Reporting System and how NRS data can be used for program 
>improvement.
> >
> >Sandy Strunk, PLI List Moderator
> >
> >********************************************************
> >
> >
> >	Several years ago, I started a new job and my first assignment was a 
>very >small project to help the Division of Adult Education and Literacy 
>(DAEL) assist >states to improve their ability to evaluate the quality of 
>local adult education >programs.  Little did I suspect that this work would 
>eventually evolve into >developing the national accountability system for 
>the adult education program, and >what we now know as the National 
>Reporting System (NRS).  Before this, I had little >knowledge of adult 
>education and accountability, but life has a way of bringing >about the 
>unexpected. >	Most people probably don't know that the NRS had its origin 
>in 1996, when >a group of state directors of adult education were the first 
>to move forward the >idea for a national accountability system.  At that 
>time there were proposals in >Congress to roll adult education into a 
>national work force development block >grant system.  It was widely feared 
>that block grants, giving governors discret!
>ion >on how to set up the system in their states, would mean the end of the 
>adult >education program.  At the same time, demands for accountability 
>increased for all >programs, at both the federal and state levels.  To 
>preserve meet these demands and >adult education, state directors and 
>others realized we needed a way to collect >valid and reliable data to show 
>what adult education is, the type of students who >enrolled and the 
>outcomes students achieved * and that these outcomes went beyond 
> >employment.  DAEL responded in 1997 with a two-year project, which I 
>directed, >to develop an outcome-ba!
> >sed accountability system in collaboration with state education directors 
> >and other adult educators.  >	One goal in developing this system was to 
>produce a set of valid and >reliable data that could demonstrate to 
>legislators and other policymakers the value >of adult education.  But we 
>also wanted the system to do more than that.  We >recognized the value of 
>data as a program improvement tool and wanted the system to be >a source of 
>information for states and local programs to use for program >management 
>and improvement. Therefore, we tried to build state flexibility into the 
>data >requirements.
> >
> >Based on the system we developed though this project in 1997-1998, the 
>NRS >was born and eventually incorporated into the Workforce Investment Act 
>(WIA) in >1998.  WIA made the system mandatory (we had considered it 
>voluntary until then) >and added a requirement for states to set 
>performance standards for outcome >measures and an incentive award program 
>to motivate performance.  Our focus then >shifted from system development 
>to implementation.
> >
> >Since 1998, we have engaged in a series of training and technical 
> >assistance activities that we hoped would build on each other to meet our 
>dual goals of >having an effective accountability system and a source of 
>data for local and state >program improvement efforts.  We planned these 
>activities in three phases.  >First, we needed to get the NRS understood 
>and working in each state, which we >accomplished through national and 
>regional training, development of policy and >procedural documents, web 
>sites and individual technical assistance. >In the second phase we focused 
>on improving the quality of data and >improving the uniformity of how 
>states and programs collected the date.  In 2002, we >produced a data 
>quality guide for local programs that described ways to collect data >in 
>valid and reliable ways and reinforced this information with regional 
> >trainings of state staff.  As these data quality improvement efforts 
>continue we moved >into the third phase of assistance, pr!
>omoting data use for program management and >improvement among states and 
>local programs.  We again developed a "how-to" >guide, accompanied by 
>regional state trainings in 2003.  Most recently, we are just >now 
>completing a third series of workshops on using data for program monitoring 
>to >assess local program performance and identify areas for change.
> >
> >Promoting NRS data use for program improvement continues to be one of our 
> >top priorities.  We see using data as the key to improving the quality of 
>local >programs and as a way to get state and local staff to understand the 
>value of data >* to see data not as an administrative burden, but a 
>valuable tool.  I think we >are beginning to see some successes in this 
>area.  We used to have to spend some >time at our trainings to answer 
>questions about the NRS and defend its purpose and >value.  Now we not only 
>hear more acceptances, but a genuine desire to learn to >use data and 
>really improve program quality�our original goal all along.  For >example, 
>programs are using data to make recruitment and enrollment decisions and 
> >schedule class times and instructional approaches.  States are using data 
>to >identify high performing and low performing programs, to target 
>technical assistance >for program improvement and make funding decisions.   
>Many states NRS data systems >now have r!
>eports built !
> >in that help local programs understand their student attendance, 
>enrollment >and outcomes and make informed decisions on program improvement 
>efforts.
> >
> >While this is heartening to know, we also know there is a lot more work 
>to >do, which brings us to our topic, using NRS data for program 
>improvement.   In >traveling the country and doing training and technical 
>assistance on this topic I >recognize some of the issues involved in using 
>data and have heard several of the >questions that can serve as a basis for 
>our discussion here.
> >
> >What kinds of decisions do you make using data?  What kinds of data do 
>you >use to steer local improvement?
> >
> >How accessible are these data to you?
> >
> >What successes have you had in using data?
> >
> >What data do you need that NRS does not collect?
> >
> >What are barriers to using data?  What other systems and structures do 
> >states and local programs need to put into place to supplement NRS data?
> >
> >Larry Condelli
> >American Institutes for Research
> >
> >
> >
> >------------------------------
> >
> >_______________________________________________
> >NIFL-PLI mailing list
> >NIFL-PLI@literacy.nifl.gov >To unsubscribe or change your subscription 
>settings, please go to >http://www.nifl.gov/mailman/listinfo/nifl-pli >
> >
> >End of NIFL-PLI Digest, Vol 8, Issue 1
> >**************************************
> >_______________________________________________
> >NIFL-PLI mailing list
> >NIFL-PLI@literacy.nifl.gov
> >To unsubscribe or change your subscription settings, please go to 
> >http://www.nifl.gov/mailman/listinfo/nifl-pli
> >
> >RFC822 header
> >-----------------------------------
> >
> > Return-Path: <nifl-pli-bounces@literacy.nifl.gov>
> > Received: from relay.iu13.org ([unix socket])
> > 	by mail.iu13.org (Cyrus v2.1.11-Lancaster-Lebanon-IU13-Mail-Services) 
> >with LMTP; Mon, 10 May 2004 15:47:38 -0400
> > X-Sieve: CMU Sieve 2.2
> > Received: from gabbro.iu13.org (gabbro.iu13.org [172.21.1.73])
> > 	by relay.iu13.org (Postfix) with ESMTP
> > 	id BC6584701CF; Mon, 10 May 2004 15:47:37 -0400 (EDT)
> > Received: from literacy.nifl.gov (unknown [192.188.111.2])
> > 	by gabbro.iu13.org (Postfix) with ESMTP
> > 	id 1A1672FEA9; Mon, 10 May 2004 15:47:35 -0400 (EDT)
> > Received: from literacy.nifl.gov (localhost [127.0.0.1])
> > 	by literacy.nifl.gov (8.10.2/8.10.2) with ESMTP id i4AJkLm12593;
> > 	Mon, 10 May 2004 15:46:21 -0400 (EDT)
> > Received: from NW51_1 (mail.southbaltimorelearns.org [141.157.37.194])
> > 	by literacy.nifl.gov (8.10.2/8.10.2) with SMTP id i4AJURm12187
> > 	for <nifl-pli@nifl.gov>; Mon, 10 May 2004 15:30:27 -0400 (EDT)
> > Received: from SBLC_DOM-Message_Server by NW51_1
> > 	with Novell_GroupWise; Mon, 10 May 2004 15:27:24 -0400
> > Message-Id: <s09f9f5c.024@NW51_1>
> > X-Mailer: Novell GroupWise 5.5.4
> > Date: Mon, 10 May 2004 15:27:15 -0400
> > From: "Sonia Socha" <Ssocha@southbaltimorelearns.org>
> > To: <nifl-pli@nifl.gov>
> > Mime-Version: 1.0
> > Content-Type: text/plain; charset=ISO-8859-7
> > Content-Disposition: inline
> > Content-Transfer-Encoding: 8bit
> > X-MIME-Autoconverted: from quoted-printable to 8bit by literacy.nifl.gov 
>id
> > 	i4AJURm12187
> > X-Mailman-Approved-At: Mon, 10 May 2004 15:46:14 -0400
> > Cc: > Subject: [NIFL-PLI] Re: NIFL-PLI Digest, Vol 8, Issue 1
> > X-BeenThere: nifl-pli@literacy.nifl.gov
> > X-Mailman-Version: 2.1.4
> > Precedence: list
> > Reply-To: Program Leadership and Improvement Discussion List 
> ><nifl-pli@nifl.gov>
> > List-Id: Program Leadership and Improvement Discussion List
> > 	<nifl-pli.literacy.nifl.gov>
> > List-Unsubscribe: <http://www.nifl.gov/mailman/listinfo/nifl-pli>,
> > 	<mailto:nifl-pli-request@literacy.nifl.gov?subject=unsubscribe>
> > List-Archive: <http://www.nifl.gov/pipermail/nifl-pli>
> > List-Post: <mailto:nifl-pli@literacy.nifl.gov>
> > List-Help: <mailto:nifl-pli-request@literacy.nifl.gov?subject=help>
> > List-Subscribe: <http://www.nifl.gov/mailman/listinfo/nifl-pli>,
> > 	<mailto:nifl-pli-request@literacy.nifl.gov?subject=subscribe>
> > Sender: nifl-pli-bounces@nifl.gov
> > Errors-To: nifl-pli-bounces@nifl.gov
> > X-Content-Scanned: by IU 13 eMail Scanning
> >
>
>_______________________________________________
>NIFL-PLI mailing list
>NIFL-PLI@literacy.nifl.gov
>To unsubscribe or change your subscription settings, please go to
>http://www.nifl.gov/mailman/listinfo/nifl-pli

_________________________________________________________________
Stop worrying about overloading your inbox - get MSN Hotmail Extra Storage! 
http://join.msn.com/?pgmarket=en-us&page=hotmail/es2&ST=1/go/onm00200362ave/direct/01/