Hi Mary,
 
Using the CASAS 5 point gains and United Way credits are good ways to
help your program get the recognition and credit for helping learners.
We encourage states and local programs to use such strategies to
demonstrate gain for small programs such as yours and others that serve
students at the lower literacy levels.  This is not a distortion at all
but a more accurate way to look at your program gains in this case.
 
The NRS is a national system and for that reason, there has to be be
some generalizations and standardization so that cross-program and
cross-state aggregation is meaningful.  But there is flexibility at the
state and local level to use other measures to show gains and progress
and to meet the accountability requirements we all have to face.  We
hope states and local programs take advantage of this flexibility at the
local level, where there is no need for national aggregation.  Your
state or local program in your area could pool your results for your own
use. 

________________________________

From: assessment-bounces at nifl.gov [mailto:assessment-bounces at nifl.gov]
On Behalf Of Mary Beheler
Sent: Tuesday, April 17, 2007 4:58 PM
To: The Assessment Discussion List
Subject: [Assessment 733] Re: What about us little guys using volunteers
forone-on-one tutoring?


Yes, WV would "consider our mitigating circumstances" if we had missed a
few of our goals. But we would also get a warning about, "Money is
tight, and if you don't make your goals...." 
 
We have made our goals, but sometimes completing the nagging and
pleading it takes to get the one last student in to post test (and
succeed) has been frighteningly close to the fiscal year deadline. We
had a group of 5. Three post-tested: 2 "improved a level" and 1 did not
(even though he had actually gained more CASAS points than the others).
Therefore, our NRS score at that level was only 40%. We needed something
above 50%. We had to get another student assessed!
 
One missing student had moved hundreds of miles away. The other had a
new job providing him many over-time hours. He finally came in on the
last possible day to post test and fell across the FFL line with a 1 or
2 point gain. Our improvement rate for that level suddenly jumped from
40 to 60%! That's silly (IMHO).
 
To complicate things a bit, other funders, such as United Way, look at
these statistics as well. Explaining levels and how some students can
improve quite a bit and not be a "success" is hard to do without
sounding like we are just whining. Fortunately, they have allowed us to
use any 5 point CASAS gain as a measure of success instead of using the
NRS brackets. And if one student comes up 20 points, we get 4 United Way
credits! (On the NRS report it counts only one gain in his entry level,
though he may have crossed more FFL lines.
 
Doesn't using using the different level brackets, instead of total
points gained by each individual, distort the results, even in larger
groups? Why are they used?
 
Since the statistics for small groups can be changed so drastically by
even one individual, why not pool the results from groups like ours?
(Call the pool "The Long Tail" if you must:
http://en.wikipedia.org/wiki/The_Long_Tail) The pool might give
researchers an idea of whether classrooms or one-on-one tutoring is most
effective at the lowest literacy levels.
 
So, if you have any influence, please try to persuade NRS to describe
the goals for small groups in terms that make more sense for our
situation. Right now we are like little kids clomping around in Mom's
high heels. It is hard to work that way, and very far from useful.
 
All of which is not to say that individual assessments are not quite
helpful for for spotting individual problems and successes. They do
help.
 
Mary G. Beheler
Tri-State Literacy
455 Ninth Street
Huntington, WV 25701
304 528-5700, ext 156 

	-----Original Message-----
	From: assessment-bounces at nifl.gov
[mailto:assessment-bounces at nifl.gov]On Behalf Of Rosemary Matt
	Sent: Tuesday, April 17, 2007 7:39 AM
	To: The Assessment Discussion List
	Subject: [Assessment 724] Re: What about us little guys
usingvolunteersforone-on-one tutoring?
	
	
	Good Morning Everyone!
	 
	I am sorry to be so late joining the group.  I am one of the
panelists, Rosemary Matt.  Just recently, I accepted the position of NRS
Liaison for New York State so monitoring data and providing technical
assistance to programs in need is now my entire focus.  We also have a
large contingent of programs that provide service through a volunteer
network in a one to one tutoring arrangement.  Your concerns Mary
regarding the inability for programs such as these to meet performance
benchmarks is shared by New York programs as well.  Our state department
has thoroughly considered the population these folks serve and consider
that to be a mitigating circumstance when assessing their performance.
The value of these organizations serving some of our lowest skilled
readers is well known and appreciated.  In a state the size of New York
it is possible to absorb the lack of educational gain increments from
these agencies as they are balanced by other programs serving students
for whom gain is eminent.  
	 
	At the same time however we have worked closely with these
programs and their statewide leadership team to provide technical
assistance in the area of assessment.  As they learn more about the
strategies and nuances that evolve around the NRS accountability system,
they are better able to show whatever gain is possible from their
students.  
	 
	As some of you are aware, New York also utilizes the program
level Report Card.  We attended the training two years ago that was
provided by Larry and his staff at AIR.  I would strongly recommend this
training to any state considering this accountability tool for programs.
We have made incredible advances in terms of identifying high performing
programs and targeting those in need of technical assistance through our
Report Card Rubric.  Marie has posted three power points that I offer in
training built around this rubric.  To further support our volunteer
programs, our state department has chosen to rank these programs among
themselves providing a homogeneous category specific to their needs.
They are not measured against the cohort of traditional adult education
programs.
	 
	Another strategy we have recently embarked upon is through our
statewide data system, we have introduced Collaboration Metrics.  Many
students working first with these volunteer programs while they are at
minimal skill levels will eventually move into traditional programming
and continue to succeed through the educational levels.  To ensure the
volunteer programs remain tied to the student's success, they are
informed of the students progress through the data system and can
subsequently report on that gain as well.  
	 
	These few methods of support have been well received by our
volunteer affiliates.  Hope they may give you and your state some
thoughts for the future.
	Rosemary
	 
	 
	 
	Rosemary I. Matt
	NRS Liaison for NYS
	Literacy Assistance Center
	12 Meadowbrook Drive
	New Hartford, NY 13413
	315.798.1026

________________________________

	From: assessment-bounces at nifl.gov on behalf of Sandy Strunk
	Sent: Mon 4/16/2007 5:58 PM
	To: The Assessment Discussion List
	Subject: [Assessment 723] Re: What about us little guys using
volunteersforone-on-one tutoring?
	
	

	Mary,

	I think you raise some valid concerns. When you're working with
a small pool of students like this, aggregate statistics can be rather
meaningless. I would think the most helpful data for you would be
individual diagnostic reading assessment and progress monitoring data.
Are you familiar with the Adult Reading Component study and the work
related to using reading profiles? You might want to check out
http://www.nifl.gov/readingprofiles/. I'm wondering if the component
reading assessments wouldn't go a long way toward "focusing" the reading
instruction you offer based on each learner's profile. That doesn't get
you off the hook for NRS reporting, but it does provide a mechanism for
meeting the highly individual needs of your learners. Just a thought.

	 

	Sandy Strunk

	Program Director for Community Education

	Lancaster-Lebanon Intermediate Unit 13

	1020 New Holland Avenue

	Lancaster, PA 17601

	(717) 606-1873

	
________________________________


	From: assessment-bounces at nifl.gov
[mailto:assessment-bounces at nifl.gov] On Behalf Of Mary Beheler
	Sent: Monday, April 16, 2007 5:03 PM
	To: The Assessment Discussion List
	Subject: [Assessment 721] Re: What about us little guys using
volunteers forone-on-one tutoring?

	 

	We tutor adults. No children.

	Almost all our students are at Begining Literacy to High
Intermediate ABE level. Almost no high or even low adult secondary. At
the secondary level we only get the students that can't (or won't)
tolerate study in a regular ABE classroom. ESL instruction is done by a
different organization, with paid teachers.

	 

	We net 20 to 25 students with more than 12 hours of study each
year. We are so small that sometimes an entire FFL will have only one
student in it. When that happens the only question is, "How many
advanced a level: 0 or 100%?"

	 

	We deal with students on a highly individualized basis. One may
need to learn to read again after having a stroke or a fever. Another
may have taught himself to sight read at a very high level, but
neglected to teach himself any spelling or writing skills. A high school
graduate may not have learned even his ABCs, for whatever reason. One or
two students a year might have an employment or higher education goal.
(Then WV can't verify it, if the student works or studies out of state.)
I can safely say that no two students have been alike in the nearly ten
years that I have been here.

	 

	I genuinely *like* statistics and know they can be very useful,
and don't mind gathering data to be put in a bigger pool if what comes
back is helpful. However, if a level has only 3 students, is the data
even "statistically significant" if just 2 of them are available for
both pre and post assessment? 2 of 4? 

	 

	Some things are better seen by microscopes and others by
telescopes. Right now neither NRS nor CASAS seems especially useful at a
local level. Maybe all I need is to find out how to focus them. Maybe
they should be just trashed. They may cost more to use than they return
in terms of time and money and *stress*, on us and especially on our
students.

	 

	I'm from West Virginia, not Missouri, but, "Show me!" (Please.)

	 

	Mary G. Beheler
	Tri-State Literacy
	455 Ninth Street
	Huntington, WV 25701
	304 528-5700, ext 156 

		-----Original Message-----
		From: assessment-bounces at nifl.gov
[mailto:assessment-bounces at nifl.gov]On Behalf Of Marie Cora
		Sent: Monday, April 16, 2007 1:29 PM
		To: Assessment at nifl.gov
		Subject: [Assessment 714] Just joining us? Here's what
you need to know...

		Hi folks,

		 

		A number of subscribers have just joined us and so I
would like to give them the necessary info for joining our discussion.
Please post your questions and share your experiences now!

		 

		View the archives at:
http://www.nifl.gov/pipermail/assessment/2007/date.html to get up to
date with the current conversation.

		 

		See suggested resources at:
http://www.nifl.gov/lincs/discussions/assessment/07program_impr.html
(Scroll to the bottom!!)

		See more resources at:  

		For the 3 power points from New York State/LAC, click on
these links:

	
http://www.nifl.gov/lincs/discussions/list_docs/ReportCardRubric.ppt

	
http://www.nifl.gov/lincs/discussions/list_docs/RollingOutReportCard.ppt

	
http://www.nifl.gov/lincs/discussions/list_docs/DevelopingDisseminatingR
eportCards.ppt

		Here are your prompts: add your voice!:

		* Do you use data in your program? What type? How? What
have been the results? 

		* What information (data) would you like to track and
why? 

		* What data would you like to learn how to use? 

		Thanks!!

		 

		Marie Cora

		 

		 

		 

		Marie Cora

		marie.cora at hotspurpartners.com
<mailto:marie.cora at hotspurpartners.com> 

		NIFL Assessment Discussion List Moderator

		http://www.nifl.gov/mailman/listinfo/assessment

		Coordinator, LINCS Assessment Special Collection

		http://literacy.kent.edu/Midwest/assessment/

		 

		 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.nifl.gov/pipermail/assessment/attachments/20070417/8b34c3ef/attachment.html