Hi, Don.  Thanks for doing this!  As a former Even Start evaluator, my
experience has been that evaluation (and sometimes evaluators) scare people
to death.  I remember teachers being incredibly nervous when I'd come into
their classrooms.  Somehow they thought that I was evaluating them rather
than the program.

 

The big message I always tried to emphasize with Even Start personnel was
that the purpose of the evaluation is program improvement.  So when you talk
about the practical aspects of evaluation, Even Start evaluation should be
nothing but practical.  If Even Start staff together with their evaluator
can identify issues within the program that need to be improved, then we've
all done our job.  But evaluation should not be only about finding
weaknesses or issues; it should also be about identifying strengths within
the program and celebrating them.

 

You also talked about data.  There is so much data available in Even Start!
But I've found programs who use an assessment and collect data only for the
purpose of reporting to the state without knowing how to collect and use the
data for informing their own program.  For instance, Texas uses the HOME
Inventory as a parenting measure.  I found lots of projects that dutifully
gave that assessment twice a year, collected the data, entered it into our
management information system, and then were done with it.  But with just a
little bit of work and an Excel spreadsheet, that very data can be used to
inform parent education, home visits and adult education programs about the
needs of individual families and the collective needs of families in the
program!  

 

And when we can show gains on standardized assessments in Even Start, that's
the kind of information policy makers want and need in order to make
informed decisions about the program on a state or national level and
certainly is the kind of information that Congress needs as they deliberate
on whether or not to restore Even Start to its previous funding level.  

 

I loved your comment about the role of evaluator in regard to advocacy.
Evaluators have the data that show Even Start's benefits to families!

 

I hope more folks with chime in on this conversation in the next few days.

 

Best,

Deborah

 

Deborah Stedman, Ph. D.

Assistant Professor, Developmental and Adult Education

Director, Texas Family Literacy Resource Center

Texas State University

601 University Drive

San Marcos, TX  78666

512-245-9757

ds43 at txstate.edu

Texas State University - San Marcos is a member of the Texas State
University System

 

 

 

  _____  

From: familyliteracy-bounces at nifl.gov
[mailto:familyliteracy-bounces at nifl.gov] On Behalf Of Ann Seaman
Sent: Tuesday, February 27, 2007 7:17 AM
To: The Family Literacy Discussion List
Subject: [FamilyLiteracy 565] initial discussion

 

Good morning.  I am Don Seaman, Professor Emeritus (although that title
really doesn't get me anything), Texas A&M University, and Former Team
Leader with the Texas Adult Education Standards project.  My wife, Anna, and
I have been independent evaluators for various family literacy projects for
a number of years and frankly, we have as many questions as we have answers
about this thing called "evaluation".  She is an early childhood education
specialist and my background is adult education, a unique combination for
working with family literacy programs.  At present, we evaluate ten family
literacy programs in Texas.

 

So, how do I approach this concept of evaluation.  First, I separate
evaluation from assessment.  "Assessment" consists of all of those things
that occur in order to acquire information and data, i.e., testing,
interviewing, observing, etc.  Up to that point, evaluation has not
occurred, even if the differences between pre-and-post-test scores have been
analyzed and observation notes have been summarized and reported.  Up to
that time, it's only assessment.  When someone finally attaches some "value"
judgment, e.g., good or not good, acceptable or not acceptable, etc, to
those findings, then, evaluation has occurred.  Remember, the word,
evaluation, is simply e-valu-ation.  Move the "e" from in front of the next
four letters to behind them and it's value-ation (sorry, Spell Check).  A
value judgment of some kind is "built in" to the word, itself.  

 

I also focus upon the "practical" aspects of evaluation - how to use it to
help all of the audiences involved in the program and I hope this is what
you are interested in discussing.  If you want to discuss theories of
evaluation, which are very important, please hold that discussion for
someone else.  I firmly believe that both qualitative and quantitative data
are important and that each can be used to effectively document the quality
of a program.  I have used data from a qualitative study as the basis to
develop an instrument with which I acquired extensive quantitative data to
document the value of a program.  I have also used findings from a broad
survey to develop and use a qualitative approach to obtain some "in-depth"
data that helped program staff make better decisions.

 

Also, please be careful of defining the term, "family literacy".  Not every
"family literacy" program contains the same kinds or numbers of components
such as those found in some nationally-funded programs and we have worked
with various kinds of "family literacy" programs without judging them in
regard how many components they contain.  For example, programs without much
financial support cannot do the same things as those that have substantial
support, but they can do "some" things and that is what we must remember in
working with the wide range of "family literacy" programs that exist.  If
"beauty" exists in the eyes of the beholder, so often does "family
literacy".

 

Therefore, because of the differences in family literacy programs, I try to
use a common approach in evaluation that can be used in almost every
situation - something that will provide the data various audiences need
while maintaining the integrity of the evaluation process.  It consists of
two major components:  (1) What outcomes are expected, and (2) for what
audiences?  Simple and easy -  right?  (Spell Check doesn't like incomplete
sentences, but we'll ignore that for now.)  (If you believe it's really that
simple and easy, I own some desert land out in West Texas I'd like to talk
to you about buying - if you believe the first proposition, you'd surely
like my sales pitch!)

 

Even within the simplified approach I use, many things complicate evaluation
activities.  The expected outcomes for a nationally-funded program, e.g.,
Barbara Bush Foundation or Even Start, may be easier to evaluate because the
expected outcomes, and audiences, may be somewhat standard across all
programs.  However, what about the smaller programs with limited funding and
which have local, specific goals, or expected outcomes?  Do those same two
evaluation concepts remain effective?   

 

They should.  For example, we (my wife and I) once evaluated a small family
literacy program whose major objective, or expected outcome, was to enroll
families and help the parents acquire their GED certificate while providing
"some educational activities" for the children.  Much of the funding was
from local, mostly private sources with very little public funding and the
emphasis was upon the parents' education and transition into the community
as "contributing members", whatever that means.  The families had limited
abilities in using English so it was essentially an ESL family literacy
program.  The program had been active for several months before were
contacted for evaluation services.  We reviewed the funded proposal and met
with the staff members who promptly informed us that there would be no GED
certificates awarded the first year because the parents' academic abilities
were much lower than expected and they would need much more time in the
program to reach their (the staff's) expectations.  Staff members were also
anxious about explaining that to the funders.  We assured them we could help
with that issue, once we had the proper documentation, i.e., initial test
scores, progress scores, etc..  

 

The children seemed to be making progress, but when we interviewed the
parents, the fun really began.  "Their" goals, or desired outcomes, were not
to acquire a GED certificate, but to "learn enough English to get a job" and
then leave the program.  To some extent, that outcome was associated with
the (employment) "transition" aspect of the program, but not with the
expected education credential.  Once we had acquired the data we needed -
assessment scores, interview data, etc., we prepared a report, but
recommended that a meeting be held at which all audiences be represented.
At the meeting, we addressed the concerns and desired outcomes from
everyone's perspective, recommending that the project continue, but with
somewhat revised expected outcomes.  We focused upon the original need for
the project (that still existed) and used our data (I can't over-emphasize
the importance of data) to demonstrate how progress was being made toward a
number of outcomes that were related to the original identified need.  We
also emphasized how "flexibility" is critical to the success of many
programs and that the needs of the participants were the overriding factor
in decision-making.  Did that take us out of an "evaluation" role into
offering technical assistance?  Maybe, but when evaluators are asked to give
their opinion about the quality of the program, i.e., a "value judgment", we
should comply.  However, that should not take us out of our role of
"independence" in the evaluation activity.

 

Again, we emphasized the expected outcomes of the various audiences, but
always remained focused upon the original need for the project, or program.
Were the needs of the participants being met?  Did we have data to verify
that?  If the expected outcomes were revised, how would that affect all of
the audiences who had a stake in the project?  In this example, the project
staff had already learned that changes in expected outcomes, especially in
the project timeline, were in order.  As all parties became aware of how
some original assumptions were not, as originally stated, addressing the
reality of the situation, everyone involved became amenable to change.  We
worked within the two principal concepts - all of the audiences, and their
expected outcomes (some of which were really not known in the beginning, a
mistake by the program staff members).  Incidentally, that project
continued.

 

I have rambled enough, but I would like to conclude this initial
presentation with some comments about the role(s) evaluators sometimes
assume in their work.  It is rare to be able to simply present the data,
make statements about how well the expected outcomes are being met, and walk
away.  People want more, and they deserve it.  We may become informants
(positively-speaking) by providing information from our experiences,
research data, or other sources that will help the staff make decisions;  we
might assume the role of mediator when differences occur among the various
audiences about the future direction of the program, using the same
aforementioned resources;  or, we may even assume some kind of advocacy role
because we perceive how the project could greatly improve the lives of the
participants who, after all, are the most important "audience" in all of
this.  Whatever we do must be done with quality work, integrity, and a
desire to foster the betterment of family literacy. 

 

I am certain that many of you, including you "lurkers", have ideas about the
evaluation process or about evaluators, themselves.  Let's hear them!  If
you have differing opinions about something I've said, or I've omitted some
good ideas, as I surely have, OK.  The purpose of this posting is to
generate discussion and share information and ideas.  That's all I've done,
or attempted to do.  If I seem to delay in responding to something, it is
simply because I am slow on the keyboard and I have learned to think about
something before responding too quickly.  Some positive things do come with
the ageing process.

 

 

Don Seaman
acs8716 at suddenlink.net

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.nifl.gov/pipermail/familyliteracy/attachments/20070227/000738df/attachment.html