Barbara,
 
I like your thinking!
 
What we advise is to monitor periodically (monthly or quarterly) program
pre-posttest rates and retention factors.  In this way you can see if a
program has had a problem in time to intervene to explore the problem
and correct it.
 
In our Data Detective training we illustrated how to use data as the
starting point for exploring what might be happening in your programs.
As you note in your example below, low educational gains may be due to
low retention, program staff not administering the posttest or students
just not making gains.  Data can help you  pinpoint which of these three
problems may exist. Once you know this, you can dig deeper to look for
the underlying issues on which you can direct program improvement (such
as changes in class schedules, better support and training for
assessment or changes in instruction that may be needed).

________________________________

From: assessment-bounces at nifl.gov [mailto:assessment-bounces at nifl.gov]
On Behalf Of bhofmeyer
Sent: Tuesday, April 17, 2007 1:30 PM
To: The Assessment Discussion List
Subject: [Assessment 730] Re: Using Data for Program Improvementbegins
today!



Larry and all,

That's true. Table 4 and Table 4b contain a great deal of information,
which if analyzed by program, site and class, can help begin to identify
strengths and areas needing additional support. For example, these two
tables can help program administrators/teachers identify some of the
following reasons behind low percentages of level gains:

1. Students aren't staying long enough to post-test. If the total
contact hours divided by the total number of students is less than the
number of hours required for post-testing, we know we have a retention
problem. Now we can look at: the intake process and transition into
class; whether instruction is meeting student needs/expectations;
whether curriculum and materials meet the varied levels and learning
styles/modalities of students; outside barriers that keep students from
attending; etc.

2. Students are staying long enough to post-test, but are not being
post-tested. (Compare the number of students on Table 4 to the number on
Table 4b.) If this is the case, we can look into whether this is a staff
problem - i.e. they have no sytem to know when a learner has enough
hours to post-test; they need assessment training/support; etc. - or a
student problem. In some programs we have found that when students learn
they will be taking a post-test, they stop coming for a period of time.
In this case, we need to help students understand that we are assessing
our effectiveness, not their intelligence.

3. Students are being post-tested, but not making level gains. (Compare
the number of students on Table 4b -i.e. those who have been post-tested
- to the number who show a gain on that table.) If this is the case, we
can begin to look at instructional strategies and curriculum/materials.

There is one other possibility that I can think of. Students may be
post-tested and making gains, but in some cases there is a glitch in the
paper flow and for some reason information is not being submitted for
data entry or data entry is flawed. If you have a system for submitting
class data back to each teacher for review, they can help you identify
if this is the case and where the glitch may be taking place.

In short, I love the NRS reports because they tell you so much about
your program if you take time to analyze them. I hope I haven't taken up
your time telling all of you things you already knew. I'm looking
forward to hearing some great new ideas this week.

Barbara Hofmeyer 
Coaching Consultant for 
Indiana DOE, Division of Adult Education 


On Tue, 17 Apr 2007 12:07:10 -0400, Condelli, Larry wrote 

> Barbara, 

>   

> That is a great idea and if you can break it down even further -- by

site or class, for example -- you can get even better insights. 

> 

> 


________________________________

From: assessment-bounces at nifl.gov [mailto:assessment-bounces at nifl.gov]
On Behalf Of bhofmeyer 

> Sent: Tuesday, April 17, 2007 9:51 AM 

> To: The Assessment Discussion List 

> Subject: [Assessment 727] Re: Using Data for Program Improvementbegins

today! 

> 

> 

> Barbara, 

> In addition to the other great ideas discussed yesterday, I'd like to

offer an additional way to analyze NRS data on the local level. If you
divide the contact hours by the number of enrolled students in each
educational functioning level (EFL) on Table 4, you will get a rough
idea of how long students in each EFL are staying in class. If you find
that students in one of the levels - say beginning literacy - are
leaving before they have enough hours to post-test or make a level gain,
you may wish to examine further the instructional strategies,
curriculum, and professional development needs of staff serving those
learners. If the average contact hours are high, but level gains in that
EFL are low, once again, a review of assesssment, instruction and
curriculum may reveal specific professional development needs for your
program. 

> Barbara Hofmeyer 

> Coaching Consultant for 

> Indiana DOE, Division of Adult Education 

> 

> On Mon, 16 Apr 2007 13:25:04 -0600, Barbara Arguedas wrote 

> > Hello Tina, 

> > I wonder if you could say more about what exact data you use to see

what your teachers need.  Is it attendance or level gain, or something
else?  

> >   

> > We use attendance data to track total hours students are attending

and to determine if they are eligible to post-test (minimum of 40 hours
in NM).  If students attend at least 75% of the potential hours, then
the student is eligible to get a certificate at the end of the session
(usually a 12-week session that meets 5 hours each week).  We also see
what the overall retention rate is by teacher as well as the post-test
rate and the level gain rate.  I agree that these could indicate a need
for training. 

> >   

> > We are interested in other ways that programs use data to help with

retention.  Thank you. 

> >   

> > Barbara Arguedas 

> > Santa Fe Community College 

> > Santa Fe, NM 

> >   

> > -----Original Message----- 

> > From: assessment-bounces at nifl.gov

[mailto:assessment-bounces at nifl.gov] On Behalf Of Tina_Luffman at yc.edu 

> > Sent: Monday, April 16, 2007 12:41 PM 

> > To: The Assessment Discussion List 

> > Subject: [Assessment 715] Re: Using Data for Program

Improvementbegins today! 

> >   

> > 

> > Hi there, 

> > 

> >   I like to use state database information to show me which teachers

are needing assistance and which teachers are modeling good practices.
The database is certainly not a final word, as we all have had groups of
students that performed well or poorly regardless of instruction.  The
data is a good place to show red flags, however. Student assessments and
staff self-assessments are also great for predicting professional
development needs. Data can also help us see what people groups we are
reaching with advertising, and which people groups are not. Then we can
create new means of recruitment for our program. 

> > 

> > Tina 

> > 

> > 

> > Tina Luffman 

> > Coordinator, Developmental Education 

> > Verde Valley Campus 

> > 928-634-6544 

> > tina_luffman at yc.edu 

> > 

> > 




> > "Condelli, Larry" <LCondelli at air.org> 

> > Sent by: assessment-bounces at nifl.gov 

> > 04/16/2007 09:15 AM 



> > Please respond to 

> > The Assessment Discussion List <assessment at nifl.gov> 	



> > 

> > To 	

> > "The Assessment Discussion List" <assessment at nifl.gov> 	



> > cc 	

> >   	



> > Subject 	

> > [Assessment 713] Re: Using Data for Program Improvement begins

today! 	


> >   



> >   	

> >   	



> > 



> > 

> > 

> > Hi Everyone, 

> >   

> > It is a pleasure to be a guest on the list this week and my thanks

to Marie for asking me and organizing this. 

> >   

> > There is a strong federal initiative to promote use of data for

program improvement at the state and level.  Through the National
Reporting system project which I direct, we have conducted several
training and technical assistance activities over the past 4 years on
this topic, including two general training seminars on using data and
more specific ones on promoting adult education programs, monitoring,
developing state and local report cards. All of the training materials
and other information on the topic, including sample work from states,
is available on the NRSWeb website, which Marie has referenced. 

> >   

> > All of the other guests have done a great deal of interesting work

and many of them attended our training (and Sandy Strunk served as a
trainer for us a few years back). 

> >   

> > I will be interested to get your questions and learn of your

experiences, as well as the responses from the other guests. 

> > 

________________________________



> > 

> > From: assessment-bounces at nifl.gov

[mailto:assessment-bounces at nifl.gov] On Behalf Of Marie Cora 

> > Sent: Monday, April 16, 2007 7:29 AM 

> > To: Assessment at nifl.gov 

> > Subject: [Assessment 710] Using Data for Program Improvement begins

today! 

> > 

> > Good morning, afternoon and evening to you all.   

> >   

> > Today begins our discussion on Using Data for Program Improvement.

I have pasted the announcement below - please note that there have been
some edits to Guest bios.  Also, I am trying to send 4 attachments (they
are power points) but I'm having a tough time getting them through the
server.  For now, you have the announcement below and as soon as I
arrange access to the Power Points, I'll let you know.  If you received
the original announcement that I sent, you have one of the attachments
already ("Using Data Effectively DCornellier").  Thank you for your
patience with this. 

> >   

> > Also, I would like to acknowledge that today is Patriot's Day and is

celebrated in many corners of the United States.  Some of our guests, as
well as subscribers, may not be present on-line with us today and this
is fine - they will catch up with us tomorrow.  For anyone ready to
begin, please feel free to post your messages. 

> >   

> > I'll start us off by asking our guests to briefly describe how they

use data in their work to improve literacy services.  Subscribers,
please post your questions and share your own experiences using data.
What type of data would you like to track and why? 

> >   

> > Thanks! 

> >   

> > Marie Cora 

> > Assessment Discussion List Moderator 

> >   ------------------------------- 

> > National Institute for Literacy 

> > Assessment mailing list 

> > Assessment at nifl.gov 

> > To unsubscribe or change your subscription settings, please go to

http://www.nifl.gov/mailman/listinfo/assessment 

> 

> -- 

> Open WebMail Project (http://openwebmail.org <http://openwebmail.org/>

) 


-- 
Open WebMail Project (http://openwebmail.org <http://openwebmail.org/> )



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.nifl.gov/pipermail/assessment/attachments/20070417/76245cd7/attachment.html