Good morning.  I am Don Seaman, Professor Emeritus (although that title really doesn't get me anything), Texas A&M University, and Former Team Leader with the Texas Adult Education Standards project.  My wife, Anna, and I have been independent evaluators for various family literacy projects for a number of years and frankly, we have as many questions as we have answers about this thing called "evaluation".  She is an early childhood education specialist and my background is adult education, a unique combination for working with family literacy programs.  At present, we evaluate ten family literacy programs in Texas.

 

So, how do I approach this concept of evaluation.  First, I separate evaluation from assessment.  "Assessment" consists of all of those things that occur in order to acquire information and data, i.e., testing, interviewing, observing, etc.  Up to that point, evaluation has not occurred, even if the differences between pre-and-post-test scores have been analyzed and observation notes have been summarized and reported.  Up to that time, it's only assessment.  When someone finally attaches some "value" judgment, e.g., good or not good, acceptable or not acceptable, etc, to those findings, then, evaluation has occurred.  Remember, the word, evaluation, is simply e-valu-ation.  Move the "e" from in front of the next four letters to behind them and it's value-ation (sorry, Spell Check).  A value judgment of some kind is "built in" to the word, itself.  

 

I also focus upon the "practical" aspects of evaluation - how to use it to help all of the audiences involved in the program and I hope this is what you are interested in discussing.  If you want to discuss theories of evaluation, which are very important, please hold that discussion for someone else.  I firmly believe that both qualitative and quantitative data are important and that each can be used to effectively document the quality of a program.  I have used data from a qualitative study as the basis to develop an instrument with which I acquired extensive quantitative data to document the value of a program.  I have also used findings from a broad survey to develop and use a qualitative approach to obtain some "in-depth" data that helped program staff make better decisions.

 

Also, please be careful of defining the term, "family literacy".  Not every "family literacy" program contains the same kinds or numbers of components such as those found in some nationally-funded programs and we have worked with various kinds of "family literacy" programs without judging them in regard how many components they contain.  For example, programs without much financial support cannot do the same things as those that have substantial support, but they can do "some" things and that is what we must remember in working with the wide range of "family literacy" programs that exist.  If "beauty" exists in the eyes of the beholder, so often does "family literacy".

 

Therefore, because of the differences in family literacy programs, I try to use a common approach in evaluation that can be used in almost every situation - something that will provide the data various audiences need while maintaining the integrity of the evaluation process.  It consists of two major components:  (1) What outcomes are expected, and (2) for what audiences?  Simple and easy -  right?  (Spell Check doesn't like incomplete sentences, but we'll ignore that for now.)  (If you believe it's really that simple and easy, I own some desert land out in West Texas I'd like to talk to you about buying - if you believe the first proposition, you'd surely like my sales pitch!)

 

Even within the simplified approach I use, many things complicate evaluation activities.  The expected outcomes for a nationally-funded program, e.g., Barbara Bush Foundation or Even Start, may be easier to evaluate because the expected outcomes, and audiences, may be somewhat standard across all programs.  However, what about the smaller programs with limited funding and which have local, specific goals, or expected outcomes?  Do those same two evaluation concepts remain effective?   

 

They should.  For example, we (my wife and I) once evaluated a small family literacy program whose major objective, or expected outcome, was to enroll families and help the parents acquire their GED certificate while providing "some educational activities" for the children.  Much of the funding was from local, mostly private sources with very little public funding and the emphasis was upon the parents' education and transition into the community as "contributing members", whatever that means.  The families had limited abilities in using English so it was essentially an ESL family literacy program.  The program had been active for several months before were contacted for evaluation services.  We reviewed the funded proposal and met with the staff members who promptly informed us that there would be no GED certificates awarded the first year because the parents' academic abilities were much lower than expected and they would need much more time in the program to reach their (the staff's) expectations.  Staff members were also anxious about explaining that to the funders.  We assured them we could help with that issue, once we had the proper documentation, i.e., initial test scores, progress scores, etc..  

 

The children seemed to be making progress, but when we interviewed the parents, the fun really began.  "Their" goals, or desired outcomes, were not to acquire a GED certificate, but to "learn enough English to get a job" and then leave the program.  To some extent, that outcome was associated with the (employment) "transition" aspect of the program, but not with the expected education credential.  Once we had acquired the data we needed - assessment scores, interview data, etc., we prepared a report, but recommended that a meeting be held at which all audiences be represented.  At the meeting, we addressed the concerns and desired outcomes from everyone's perspective, recommending that the project continue, but with somewhat revised expected outcomes.  We focused upon the original need for the project (that still existed) and used our data (I can't over-emphasize the importance of data) to demonstrate how progress was being made toward a number of outcomes that were related to the original identified need.  We also emphasized how "flexibility" is critical to the success of many programs and that the needs of the participants were the overriding factor in decision-making.  Did that take us out of an "evaluation" role into offering technical assistance?  Maybe, but when evaluators are asked to give their opinion about the quality of the program, i.e., a "value judgment", we should comply.  However, that should not take us out of our role of "independence" in the evaluation activity.

 

Again, we emphasized the expected outcomes of the various audiences, but always remained focused upon the original need for the project, or program.  Were the needs of the participants being met?  Did we have data to verify that?  If the expected outcomes were revised, how would that affect all of the audiences who had a stake in the project?  In this example, the project staff had already learned that changes in expected outcomes, especially in the project timeline, were in order.  As all parties became aware of how some original assumptions were not, as originally stated, addressing the reality of the situation, everyone involved became amenable to change.  We worked within the two principal concepts - all of the audiences, and their expected outcomes (some of which were really not known in the beginning, a mistake by the program staff members).  Incidentally, that project continued.

 

I have rambled enough, but I would like to conclude this initial presentation with some comments about the role(s) evaluators sometimes assume in their work.  It is rare to be able to simply present the data, make statements about how well the expected outcomes are being met, and walk away.  People want more, and they deserve it.  We may become informants (positively-speaking) by providing information from our experiences, research data, or other sources that will help the staff make decisions;  we might assume the role of mediator when differences occur among the various audiences about the future direction of the program, using the same aforementioned resources;  or, we may even assume some kind of advocacy role because we perceive how the project could greatly improve the lives of the participants who, after all, are the most important "audience" in all of this.  Whatever we do must be done with quality work, integrity, and a desire to foster the betterment of family literacy. 

 

I am certain that many of you, including you "lurkers", have ideas about the evaluation process or about evaluators, themselves.  Let's hear them!  If you have differing opinions about something I've said, or I've omitted some good ideas, as I surely have, OK.  The purpose of this posting is to generate discussion and share information and ideas.  That's all I've done, or attempted to do.  If I seem to delay in responding to something, it is simply because I am slow on the keyboard and I have learned to think about something before responding too quickly.  Some positive things do come with the ageing process.

 

 

Don Seaman
acs8716 at suddenlink.net
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.nifl.gov/pipermail/familyliteracy/attachments/20070227/657fd32b/attachment.html