Hi Don and Ann - I am a self-confessed lurker on this list-serv. I was
involved over a period of about four years of a family literacy programme
here in Auckland. For their web-site, including the last evaluation report
see www.comet.org.nz <http://www.comet.org.nz/>  

 

A simple point I would like to add is that evaluators need to be aware of
outcomes that are not related to those stated for the programme. These
unintended outcomes of course can be either positive or negative, but are
important to document to understand the programme in its entirety. This is
consistent with Michael Scriven's concept of 'goal-free evaluation'. An
example of an unanticipated outcome in the COMET programme that I evaluated
was the closer relationship that developed between the early childhood
centre and the primary (elementary) school as a result of the being involved
in the programme together. While in two cases, these institutions actually
shared the same physical site, the staff at both places said that they did
not know each other, never worked together etc prior to the family literacy
programme. As a result of the programme, they now do joint planning, have
joint events, co-ordinate transition between them etc. None of which was
originally planned or documented in planning documents. Positive outcomes,
but not anticipated.

 

John

 

[please note change of address details below]

 

John Benseman PhD

 

Director of Research & Evaluation, Upskilling NZ

(based at Department of Labour)

DDI at DoL 04 915 4589 - Cell: 027 454 0683

 

*   john.benseman at criticalinsight.co.nz

*   0064 9 627 4415

7 0064 9 627 4418

 

  _____  

From: familyliteracy-bounces at nifl.gov
[mailto:familyliteracy-bounces at nifl.gov] On Behalf Of Gail Price
Sent: Wednesday, 28 February 2007 5:54 a.m.
To: The Family Literacy Discussion List
Subject: [FamilyLiteracy 566] Re: initial discussion

 

The following message is posted on behalf of Lorena Gonzalez in response to
Dr. Seaman's initial post.

 

-----Original Message----- 
From: Lorena Gonzalez [mailto:LorenaGonzalez at nisd.net] 
Sent: Tue 2/27/2007 9:27 AM 
To: The Family Literacy Discussion List; acs8716 at suddenlink.net 
Cc: 
Subject: Re: [FamilyLiteracy 565] initial discussion

Dr. Seaman,
Good morning, I am curious to know how do you evaluate the ILA (PACT)
component of Even Start programs. What are the desired outcomes that you
use and what are the best approaches, in your experience, to achieve
those outcomes.
Thank you in advance.

Lorena Gonzalez
Even Start Program
(210) 678-2989

 

 

Gail J. Price

Multimedia Specialist

National Center for Family Literacy

325 W. Main Street, Suite 300

Louisville, KY 40202

gprice at famlit.org

502 584-1133, ext. 112

 

  _____  

From: familyliteracy-bounces at nifl.gov
[mailto:familyliteracy-bounces at nifl.gov] On Behalf Of Ann Seaman
Sent: Tuesday, February 27, 2007 8:17 AM
To: The Family Literacy Discussion List
Subject: [FamilyLiteracy 565] initial discussion

 

Good morning.  I am Don Seaman, Professor Emeritus (although that title
really doesn't get me anything), Texas A&M University, and Former Team
Leader with the Texas Adult Education Standards project.  My wife, Anna, and
I have been independent evaluators for various family literacy projects for
a number of years and frankly, we have as many questions as we have answers
about this thing called "evaluation".  She is an early childhood education
specialist and my background is adult education, a unique combination for
working with family literacy programs.  At present, we evaluate ten family
literacy programs in Texas.

 

So, how do I approach this concept of evaluation.  First, I separate
evaluation from assessment.  "Assessment" consists of all of those things
that occur in order to acquire information and data, i.e., testing,
interviewing, observing, etc.  Up to that point, evaluation has not
occurred, even if the differences between pre-and-post-test scores have been
analyzed and observation notes have been summarized and reported.  Up to
that time, it's only assessment.  When someone finally attaches some "value"
judgment, e.g., good or not good, acceptable or not acceptable, etc, to
those findings, then, evaluation has occurred.  Remember, the word,
evaluation, is simply e-valu-ation.  Move the "e" from in front of the next
four letters to behind them and it's value-ation (sorry, Spell Check).  A
value judgment of some kind is "built in" to the word, itself.  

 

I also focus upon the "practical" aspects of evaluation - how to use it to
help all of the audiences involved in the program and I hope this is what
you are interested in discussing.  If you want to discuss theories of
evaluation, which are very important, please hold that discussion for
someone else.  I firmly believe that both qualitative and quantitative data
are important and that each can be used to effectively document the quality
of a program.  I have used data from a qualitative study as the basis to
develop an instrument with which I acquired extensive quantitative data to
document the value of a program.  I have also used findings from a broad
survey to develop and use a qualitative approach to obtain some "in-depth"
data that helped program staff make better decisions.

 

Also, please be careful of defining the term, "family literacy".  Not every
"family literacy" program contains the same kinds or numbers of components
such as those found in some nationally-funded programs and we have worked
with various kinds of "family literacy" programs without judging them in
regard how many components they contain.  For example, programs without much
financial support cannot do the same things as those that have substantial
support, but they can do "some" things and that is what we must remember in
working with the wide range of "family literacy" programs that exist.  If
"beauty" exists in the eyes of the beholder, so often does "family
literacy".

 

Therefore, because of the differences in family literacy programs, I try to
use a common approach in evaluation that can be used in almost every
situation - something that will provide the data various audiences need
while maintaining the integrity of the evaluation process.  It consists of
two major components:  (1) What outcomes are expected, and (2) for what
audiences?  Simple and easy -  right?  (Spell Check doesn't like incomplete
sentences, but we'll ignore that for now.)  (If you believe it's really that
simple and easy, I own some desert land out in West Texas I'd like to talk
to you about buying - if you believe the first proposition, you'd surely
like my sales pitch!)

 

Even within the simplified approach I use, many things complicate evaluation
activities.  The expected outcomes for a nationally-funded program, e.g.,
Barbara Bush Foundation or Even Start, may be easier to evaluate because the
expected outcomes, and audiences, may be somewhat standard across all
programs.  However, what about the smaller programs with limited funding and
which have local, specific goals, or expected outcomes?  Do those same two
evaluation concepts remain effective?   

 

They should.  For example, we (my wife and I) once evaluated a small family
literacy program whose major objective, or expected outcome, was to enroll
families and help the parents acquire their GED certificate while providing
"some educational activities" for the children.  Much of the funding was
from local, mostly private sources with very little public funding and the
emphasis was upon the parents' education and transition into the community
as "contributing members", whatever that means.  The families had limited
abilities in using English so it was essentially an ESL family literacy
program.  The program had been active for several months before were
contacted for evaluation services.  We reviewed the funded proposal and met
with the staff members who promptly informed us that there would be no GED
certificates awarded the first year because the parents' academic abilities
were much lower than expected and they would need much more time in the
program to reach their (the staff's) expectations.  Staff members were also
anxious about explaining that to the funders.  We assured them we could help
with that issue, once we had the proper documentation, i.e., initial test
scores, progress scores, etc..  

 

The children seemed to be making progress, but when we interviewed the
parents, the fun really began.  "Their" goals, or desired outcomes, were not
to acquire a GED certificate, but to "learn enough English to get a job" and
then leave the program.  To some extent, that outcome was associated with
the (employment) "transition" aspect of the program, but not with the
expected education credential.  Once we had acquired the data we needed -
assessment scores, interview data, etc., we prepared a report, but
recommended that a meeting be held at which all audiences be represented.
At the meeting, we addressed the concerns and desired outcomes from
everyone's perspective, recommending that the project continue, but with
somewhat revised expected outcomes.  We focused upon the original need for
the project (that still existed) and used our data (I can't over-emphasize
the importance of data) to demonstrate how progress was being made toward a
number of outcomes that were related to the original identified need.  We
also emphasized how "flexibility" is critical to the success of many
programs and that the needs of the participants were the overriding factor
in decision-making.  Did that take us out of an "evaluation" role into
offering technical assistance?  Maybe, but when evaluators are asked to give
their opinion about the quality of the program, i.e., a "value judgment", we
should comply.  However, that should not take us out of our role of
"independence" in the evaluation activity.

 

Again, we emphasized the expected outcomes of the various audiences, but
always remained focused upon the original need for the project, or program.
Were the needs of the participants being met?  Did we have data to verify
that?  If the expected outcomes were revised, how would that affect all of
the audiences who had a stake in the project?  In this example, the project
staff had already learned that changes in expected outcomes, especially in
the project timeline, were in order.  As all parties became aware of how
some original assumptions were not, as originally stated, addressing the
reality of the situation, everyone involved became amenable to change.  We
worked within the two principal concepts - all of the audiences, and their
expected outcomes (some of which were really not known in the beginning, a
mistake by the program staff members).  Incidentally, that project
continued.

 

I have rambled enough, but I would like to conclude this initial
presentation with some comments about the role(s) evaluators sometimes
assume in their work.  It is rare to be able to simply present the data,
make statements about how well the expected outcomes are being met, and walk
away.  People want more, and they deserve it.  We may become informants
(positively-speaking) by providing information from our experiences,
research data, or other sources that will help the staff make decisions;  we
might assume the role of mediator when differences occur among the various
audiences about the future direction of the program, using the same
aforementioned resources;  or, we may even assume some kind of advocacy role
because we perceive how the project could greatly improve the lives of the
participants who, after all, are the most important "audience" in all of
this.  Whatever we do must be done with quality work, integrity, and a
desire to foster the betterment of family literacy. 

 

I am certain that many of you, including you "lurkers", have ideas about the
evaluation process or about evaluators, themselves.  Let's hear them!  If
you have differing opinions about something I've said, or I've omitted some
good ideas, as I surely have, OK.  The purpose of this posting is to
generate discussion and share information and ideas.  That's all I've done,
or attempted to do.  If I seem to delay in responding to something, it is
simply because I am slow on the keyboard and I have learned to think about
something before responding too quickly.  Some positive things do come with
the ageing process.

 

 

Don Seaman
acs8716 at suddenlink.net

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.nifl.gov/pipermail/familyliteracy/attachments/20070228/f302d97e/attachment.html