Thanks to Dr. Don Seaman for getting our discussion about evaluation off
to such a great start. And thanks to all of you who have responded.
Because I don't work in a program, I am always interested in hearing
from those of you who do. You have insight into the "real" world.
Remember that Dr. Seaman will be with us throughout the day today, so
please if you questions about evaluation, comments about your
experiences, or responses to those who have already shared, we all will
be glad to hear from you. That's what the List is all about.

 

Have a good day!

 

 

Gail J. Price

Multimedia Specialist

National Center for Family Literacy

325 W. Main Street, Suite 300

Louisville, KY 40202

gprice at famlit.org

502 584-1133, ext. 112

 

________________________________

From: familyliteracy-bounces at nifl.gov
[mailto:familyliteracy-bounces at nifl.gov] On Behalf Of John Benseman
Sent: Tuesday, February 27, 2007 2:34 PM
To: 'The Family Literacy Discussion List'
Subject: [FamilyLiteracy 570] Re: initial discussion

 

Hi Don and Ann - I am a self-confessed lurker on this list-serv. I was
involved over a period of about four years of a family literacy
programme here in Auckland. For their web-site, including the last
evaluation report see www.comet.org.nz <http://www.comet.org.nz/>  

 

A simple point I would like to add is that evaluators need to be aware
of outcomes that are not related to those stated for the programme.
These unintended outcomes of course can be either positive or negative,
but are important to document to understand the programme in its
entirety. This is consistent with Michael Scriven's concept of
'goal-free evaluation'. An example of an unanticipated outcome in the
COMET programme that I evaluated was the closer relationship that
developed between the early childhood centre and the primary
(elementary) school as a result of the being involved in the programme
together. While in two cases, these institutions actually shared the
same physical site, the staff at both places said that they did not know
each other, never worked together etc prior to the family literacy
programme. As a result of the programme, they now do joint planning,
have joint events, co-ordinate transition between them etc. None of
which was originally planned or documented in planning documents.
Positive outcomes, but not anticipated.

 

John

 

[please note change of address details below]

 

John Benseman PhD

 

Director of Research & Evaluation, Upskilling NZ

(based at Department of Labour)

DDI at DoL 04 915 4589 - Cell: 027 454 0683

 

*   john.benseman at criticalinsight.co.nz

*   0064 9 627 4415

7 0064 9 627 4418

 

________________________________

From: familyliteracy-bounces at nifl.gov
[mailto:familyliteracy-bounces at nifl.gov] On Behalf Of Gail Price
Sent: Wednesday, 28 February 2007 5:54 a.m.
To: The Family Literacy Discussion List
Subject: [FamilyLiteracy 566] Re: initial discussion

 

The following message is posted on behalf of Lorena Gonzalez in response
to Dr. Seaman's initial post.

 

-----Original Message----- 
From: Lorena Gonzalez [mailto:LorenaGonzalez at nisd.net] 
Sent: Tue 2/27/2007 9:27 AM 
To: The Family Literacy Discussion List; acs8716 at suddenlink.net 
Cc: 
Subject: Re: [FamilyLiteracy 565] initial discussion

Dr. Seaman,
Good morning, I am curious to know how do you evaluate the ILA (PACT)
component of Even Start programs. What are the desired outcomes that you
use and what are the best approaches, in your experience, to achieve
those outcomes.
Thank you in advance.

Lorena Gonzalez
Even Start Program
(210) 678-2989

 

 

Gail J. Price

Multimedia Specialist

National Center for Family Literacy

325 W. Main Street, Suite 300

Louisville, KY 40202

gprice at famlit.org

502 584-1133, ext. 112

 

________________________________

From: familyliteracy-bounces at nifl.gov
[mailto:familyliteracy-bounces at nifl.gov] On Behalf Of Ann Seaman
Sent: Tuesday, February 27, 2007 8:17 AM
To: The Family Literacy Discussion List
Subject: [FamilyLiteracy 565] initial discussion

 

Good morning.  I am Don Seaman, Professor Emeritus (although that title
really doesn't get me anything), Texas A&M University, and Former Team
Leader with the Texas Adult Education Standards project.  My wife, Anna,
and I have been independent evaluators for various family literacy
projects for a number of years and frankly, we have as many questions as
we have answers about this thing called "evaluation".  She is an early
childhood education specialist and my background is adult education, a
unique combination for working with family literacy programs.  At
present, we evaluate ten family literacy programs in Texas.

 

So, how do I approach this concept of evaluation.  First, I separate
evaluation from assessment.  "Assessment" consists of all of those
things that occur in order to acquire information and data, i.e.,
testing, interviewing, observing, etc.  Up to that point, evaluation has
not occurred, even if the differences between pre-and-post-test scores
have been analyzed and observation notes have been summarized and
reported.  Up to that time, it's only assessment.  When someone finally
attaches some "value" judgment, e.g., good or not good, acceptable or
not acceptable, etc, to those findings, then, evaluation has occurred.
Remember, the word, evaluation, is simply e-valu-ation.  Move the "e"
from in front of the next four letters to behind them and it's
value-ation (sorry, Spell Check).  A value judgment of some kind is
"built in" to the word, itself.  

 

I also focus upon the "practical" aspects of evaluation - how to use it
to help all of the audiences involved in the program and I hope this is
what you are interested in discussing.  If you want to discuss theories
of evaluation, which are very important, please hold that discussion for
someone else.  I firmly believe that both qualitative and quantitative
data are important and that each can be used to effectively document the
quality of a program.  I have used data from a qualitative study as the
basis to develop an instrument with which I acquired extensive
quantitative data to document the value of a program.  I have also used
findings from a broad survey to develop and use a qualitative approach
to obtain some "in-depth" data that helped program staff make better
decisions.

 

Also, please be careful of defining the term, "family literacy".  Not
every "family literacy" program contains the same kinds or numbers of
components such as those found in some nationally-funded programs and we
have worked with various kinds of "family literacy" programs without
judging them in regard how many components they contain.  For example,
programs without much financial support cannot do the same things as
those that have substantial support, but they can do "some" things and
that is what we must remember in working with the wide range of "family
literacy" programs that exist.  If "beauty" exists in the eyes of the
beholder, so often does "family literacy".

 

Therefore, because of the differences in family literacy programs, I try
to use a common approach in evaluation that can be used in almost every
situation - something that will provide the data various audiences need
while maintaining the integrity of the evaluation process.  It consists
of two major components:  (1) What outcomes are expected, and (2) for
what audiences?  Simple and easy -  right?  (Spell Check doesn't like
incomplete sentences, but we'll ignore that for now.)  (If you believe
it's really that simple and easy, I own some desert land out in West
Texas I'd like to talk to you about buying - if you believe the first
proposition, you'd surely like my sales pitch!)

 

Even within the simplified approach I use, many things complicate
evaluation activities.  The expected outcomes for a nationally-funded
program, e.g., Barbara Bush Foundation or Even Start, may be easier to
evaluate because the expected outcomes, and audiences, may be somewhat
standard across all programs.  However, what about the smaller programs
with limited funding and which have local, specific goals, or expected
outcomes?  Do those same two evaluation concepts remain effective?   

 

They should.  For example, we (my wife and I) once evaluated a small
family literacy program whose major objective, or expected outcome, was
to enroll families and help the parents acquire their GED certificate
while providing "some educational activities" for the children.  Much of
the funding was from local, mostly private sources with very little
public funding and the emphasis was upon the parents' education and
transition into the community as "contributing members", whatever that
means.  The families had limited abilities in using English so it was
essentially an ESL family literacy program.  The program had been active
for several months before were contacted for evaluation services.  We
reviewed the funded proposal and met with the staff members who promptly
informed us that there would be no GED certificates awarded the first
year because the parents' academic abilities were much lower than
expected and they would need much more time in the program to reach
their (the staff's) expectations.  Staff members were also anxious about
explaining that to the funders.  We assured them we could help with that
issue, once we had the proper documentation, i.e., initial test scores,
progress scores, etc..  

 

The children seemed to be making progress, but when we interviewed the
parents, the fun really began.  "Their" goals, or desired outcomes, were
not to acquire a GED certificate, but to "learn enough English to get a
job" and then leave the program.  To some extent, that outcome was
associated with the (employment) "transition" aspect of the program, but
not with the expected education credential.  Once we had acquired the
data we needed - assessment scores, interview data, etc., we prepared a
report, but recommended that a meeting be held at which all audiences be
represented.  At the meeting, we addressed the concerns and desired
outcomes from everyone's perspective, recommending that the project
continue, but with somewhat revised expected outcomes.  We focused upon
the original need for the project (that still existed) and used our data
(I can't over-emphasize the importance of data) to demonstrate how
progress was being made toward a number of outcomes that were related to
the original identified need.  We also emphasized how "flexibility" is
critical to the success of many programs and that the needs of the
participants were the overriding factor in decision-making.  Did that
take us out of an "evaluation" role into offering technical assistance?
Maybe, but when evaluators are asked to give their opinion about the
quality of the program, i.e., a "value judgment", we should comply.
However, that should not take us out of our role of "independence" in
the evaluation activity.

 

Again, we emphasized the expected outcomes of the various audiences, but
always remained focused upon the original need for the project, or
program.  Were the needs of the participants being met?  Did we have
data to verify that?  If the expected outcomes were revised, how would
that affect all of the audiences who had a stake in the project?  In
this example, the project staff had already learned that changes in
expected outcomes, especially in the project timeline, were in order.
As all parties became aware of how some original assumptions were not,
as originally stated, addressing the reality of the situation, everyone
involved became amenable to change.  We worked within the two principal
concepts - all of the audiences, and their expected outcomes (some of
which were really not known in the beginning, a mistake by the program
staff members).  Incidentally, that project continued.

 

I have rambled enough, but I would like to conclude this initial
presentation with some comments about the role(s) evaluators sometimes
assume in their work.  It is rare to be able to simply present the data,
make statements about how well the expected outcomes are being met, and
walk away.  People want more, and they deserve it.  We may become
informants (positively-speaking) by providing information from our
experiences, research data, or other sources that will help the staff
make decisions;  we might assume the role of mediator when differences
occur among the various audiences about the future direction of the
program, using the same aforementioned resources;  or, we may even
assume some kind of advocacy role because we perceive how the project
could greatly improve the lives of the participants who, after all, are
the most important "audience" in all of this.  Whatever we do must be
done with quality work, integrity, and a desire to foster the betterment
of family literacy. 

 

I am certain that many of you, including you "lurkers", have ideas about
the evaluation process or about evaluators, themselves.  Let's hear
them!  If you have differing opinions about something I've said, or I've
omitted some good ideas, as I surely have, OK.  The purpose of this
posting is to generate discussion and share information and ideas.
That's all I've done, or attempted to do.  If I seem to delay in
responding to something, it is simply because I am slow on the keyboard
and I have learned to think about something before responding too
quickly.  Some positive things do come with the ageing process.

 

 

Don Seaman
acs8716 at suddenlink.net

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.nifl.gov/pipermail/familyliteracy/attachments/20070228/963155b0/attachment.html