Ton Sticht's reply reminds us never to say never.....research HAS been done.  But the question that popped into my head as I read David's comments and the responses is just who or what is being held accountable?  The research suggested by David would focus on teacher accountability as measured through how much the training impacted them and their learners or how much the teachers adhered to--and I do mean ADHERED TO--the desired outcomes of the training.  Is that accountability for the training/trainers or the trainees?  

As I may have mentioned once before on this listserv, one of my doctoral colleagues at Lesly U did very interesting dissertation research on what exactly trainees take away from training.  She herself was a highly trained, rigorously supervised pedagogy trainer for a private company in MA which does extensive PD work with K-12 teachers.  My colleague wondered if what the trainees took away from a highly structured, well-tried and well-received training program was what the training company wanted and expected.  Not surprisingly, since these are humans we are talking about, what she found was that what trainees took away was very different from person to person.  What teachers took away was highly influenced by several factors including how long the teacher had taught, how the teacher perceived him or herself as a teacher, what his or her core, deep-seated philosophy of teaching and learning was, and what the teacher needed most in his or her current teaching environment.   

These are some rather significant variables.  As a trainer, I ran head on into these variables not long ago when training a group of adult ESOL practitioners.  Despite the fact that all had received the same basic training and read the same basic materials, what the teachers were able to relate to and take out of the training to impact their teaching was extremely different and the differences could be explained according to the variables explored in my friend's research.  Two teachers who chose the most extreme change in practice both had the educational background and teaching experience to know that what they were doing previously was essentially ineffective--their words, not mine--and they had the ability to reflect pedagogically on what change would mean and how to achieve it.   Others had none of that background --one of those with virtually no ESL experience had no way of connecting what she knew to what she perceived was expected of her and she has required a good deal of follow-up to support changes in her practice.  Another with no educational training in pedagogy was one of those for whom the core beliefs in teaching and learning were deeply embedded--and deeply in conflict with values she was being asked to ponder.    Changes in her practice have been superficial--but there have been changes.  

With that research and my own experience in mind, I can't help but question what exactly accountability would entail for PD.  As I said in the opening-- is it the teacher's level of change that we want to measure or the fact the PD trainer did what he or she said or was engaged to do?  And what was that, anyway?   In my own experience as a
trainee in PD, I found that the outcomes that were stated and promised
had far more to do with learning that took place in the training
session; behavioral changes were more or less hoped for but not
measurable: "Participants will be able to implement this practice in
their classrooms," or " Participants will understand how to use this
tool to measure their learner's progress."  

What David proposes would be suitable for trainees in a proprietary training program--for example the Wilson system---where all were expected to emerge with a similar set of skills and then were measured in their ability to  apply the skills. However, to my knowledge, even Wilson trainees are not held accountable by the Wilson training program for how well their students learn to read!!  This is what, in effect, David is suggesting: That a specific PD undertaking would be rated by how well learners of teachers trained in a specific PD project would do.  

(This also makes me think of an article I read a couple of years ago about the impact of teacher expectation of students on the outcome of such a training endeavor.  Two science teachers in a KY high school were given identical training in implementing a new science curriculum--the intent of the inquiry about how they implemented it did not have as its original intent to compare the two or compare them to a control--but when the results of the two classes were compared --both with students drawn from similar backgrounds--one teacher's class did wonderfully--beyond what the company providing the training in the curriculum expected.  The other teacher's class did very poorly.  The only factor that could be identified to explain this difference was that the first teacher communicated to her students that they would do very well using this curriculum and she was excited to be learning to use it, while the second communicated almost exactly the opposite:  She was unhappy to be forced to use the curriculum and was sure her students would learn nothing from it.  Mind you, the actual day-to-day implementation was monitored, so no differences could be found there.  Again, there is that pesky human factor in trying to herd teachers into better or different practice.....)

Having had that experience that my colleague found in her research, I now set my own personal measure of accountability:  teachers in my projects MUST have tried SOMETHING that would change their practice and must reflect on that process and on outcomes of any kind.   I find that simply requiring engagement without prescribing-- or indeed PROscribing--the nature and depth of that engagement causes a great deal of reflection regardless of what actual practice the teacher has decided to try out or try changing.   Thus, my goal and that stated to those who employ me, of impacting practice--with no promises as to how much-- is measured only in terms of teacher engagement in their personally chosen project, and not by learner change.  Learner change is inevitably an outcome of the teacher's changing his or her practice--and my explicit purpose in providing the training is ultimately to improve learner outcomes-- but the extent of that change is left up to institutions or programs to measure.  I make no promises about the extent of that change.  If I did , I would be setting myself up for failure.  

And then there are the overwhelming factors of current adult ed/ESOL program practice: open enrollment and one-room-for all- instruction, to say nothing of free classes.  How does one factor OUT those variables in asking teachers to be more effective?   

And though I am supposed to be doing other research and not blabbing away on the listservs, I will stick in one other comment-- did anyone see that report coming out of one of the Carolinas a year or more ago where learner outcomes of teachers who had extensive PD were compared to those of teachers with almost no PD--and no statistically significant difference could be found?  

Sometimes we achieve outcomes we desired but that fall short of the REAL desired outcomes:  An experiment with an orientation week for an adult ESOL program was tried in a large western state in the hopes of improving learner retention-- it improved learner retention almost unbelievably--but virtually no statistical difference in learning outcomes could be measured between the experimental group--those who had the orientation--and the control---those who did not.  Since there was a belief that more hours in chairs will result in more learning, what was REALLY wanted from the orientation effort was better learning outcomes. Nonetheless, it DID meet its stated outcomes--increased learner retention.  
 
So I would urge extreme caution and plenty of reflection on what it is should be measured in accountability and how it would be measured. But do not mistake me-- I am the first to believe that trainers and PD efforts SHOULD be held accountable in some way.  Only we need to look closely at what is being promised in the PD as an outcome--Is learner change being promised o teacher change or program change?  How would the trainers then propose that that be measured, and how would trainers account for the differences I noted above about teachers being able to take professional advantage of or profit from training?  

Longwindedly yours,  Robin Lovrien Schwarz


-----Original Message-----





From: David J. Rosen <djrosen at comcast.net>
To: The Adult Literacy Professional Development Discussion List <professionaldevelopment at nifl.gov>
Sent: Mon, 7 Jan 2008 8:36 am
Subject: [ProfessionalDevelopment 1831] Is Professional Development accountable?










Jackie and others,

Since you have opened up the discussion more broadly to professional  
development and accountability in general, here are my thoughts:

At the risk of offending some people, I'll speak my mind on this  
issue.  I don't think we have good accountability for professional  
development in adult literacy education and I don't see that we soon  
will. I believe what we often have is the worst kind of "make-do"  
accountability.  There are reasons, sad though they are, for this  
predicament.

True professional development accountability would be built on what  
we know about good practice from many classroom studies. This  
research would involve collecting baseline data on what participating  
teachers know and do in the classroom, and how their students  
perform. It would involve a professional development training  
intervention under conditions that would have as few confounding  
variables as possible -- it's not an easy task to arrange such an  
intervention in education. The studies would include control groups  
with teachers who did not have the training. The interventions would  
be specific sets of methods, techniques or approaches, ones for which  
we already have qualitative or other evidence that there is  
likelihood that the change(s) in practice would positively affect  
student learning. The research would have ongoing observations to  
determine how the intervention, the new methods, techniques or  
approaches, was actually being implemented in the classroom, and it  
would require post-assessment of learners and teachers to determine  
changes in teaching and learning practices and learning outcomes.  
There would be many studies of this kind on which to base decisions  
about good practice, and then to hold teachers and administrators  
accountable for providing good practice that leads to positive  
learner outcomes, and professional developers accountable for  
providing effective professional development.

Does this sound like experimental design research? When was the last  
time we had experimental design research in adult literacy education  
professional development? I cannot cite a single example. Can you?   
Why don't we have this research? The simple truth is, it's expensive,  
and Congress has demonstrated, through the elimination of funding for  
a national center for adult literacy research that it does not care  
to spend money on research for adult literacy education.   
Furthermore, even if this were to change with a new administration  
and Congress, the chances of doing these kinds of (expensive)  
professional development studies in adult literacy education are slim.

Meanwhile, we have "make-do" accountability. In way too many states  
professional development providers and/or teachers are asked to keep  
track of and report the number of hours they offer or participate in  
professional development, or practitioners are asked to accumulate  
graduate credits, continuing education units or professional  
development points. I understand why these requirements are in place,  
but I know of no evidence in adult literacy education that shows that  
any of these bean counting strategies result in improved teacher  
performance in the classroom, not to mention improved learner outcomes.

It makes much more sense to me, if we want professional development  
accountability, not to count hours or activities or credits, but  to  
define a set of practitioner standards, including content knowledge  
and teaching skills, and to ask teachers and other practitioners to  
show they are making progress in reaching these standards. Teachers  
would use an individual professional development plan and a variety  
of resources and activities and performance assessments, to meet  
these standards. Professional developers' accountability would be  
linked to their effectiveness in helping teachers reach the  
standards.  It might also make sense to set aside paid professional  
development time to enable teachers to work on attaining these  
standards, however,  as professional development accountability needs  
to be performance-based, not based on participation in activities, a  
better compensation model might instead be to pay for demonstrated  
attainment of new competencies or standards in salary increments. Or  
both.

Some professional developers, doing the best they can with the meager  
resources they have, conscientiously collect baseline teacher and  
student data, provide well-designed interventions, and collect post  
intervention student and teacher data. This is good, the best we can  
do for now, and perhaps over a long period of time we can acquire  
enough evidence to persuade Congress that they should invest in gold  
standard adult literacy education research in areas where there is  
such evidence. We should support more of these heroic professional  
development research efforts instead of mindless requirements to  
count hours, credits, and certificates that at best are annoying and  
at worst take time away from teachers' and professional developers'  
meaningful work.

What do you think? Do agree or disagree? Speak your mind.

David J. Rosen
djrosen at comcast.net



On Jan 6, 2008, at 8:52 PM, Taylor, Jackie wrote:


> Good day or evening, all!

>

> Recently, Cristine Smith circulated a survey about “Accountability  

> and Professional Development” – specifically, the survey asks  

> professional development (PD) providers if PD related to assessment  

> and accountability training was required before the advent of the  

> National Reporting System (NRS), in the first two years after the  

> NRS, and now. It asks to what extent PD providers think that the  

> accountability system has affected the types of professional  

> development provided in their state, and to comment on what the  

> effect has been (http://www.surveymonkey.com/s.aspx? 

> sm=TseraqLIt9aH5FJ5TV3ENQ_3d_3d).

>

>

>

> These questions have gotten me to thinking about accountability in  

> general, which brings me to a new question for the group.

>

>

>

> As noted a while back on this list (http://wiki.literacytent.org/ 

> index.php/Accountability) accountability might be viewed as a  

> process that starts and ends with learning; a kind of multilane  

> highway where all parties involved have levels of accountability  

> that they should in fact meet – including the learner.

>

>

>

> Adult literacy professional development provides accountability  

> systems training to help some teachers and programs “be  

> accountable” to funders and presumably to learners. In essence, we  

> have accountability in professional development, but do we have  

> accountability _for_ professional development?

>

>

>

> Is professional development accountable for teacher and student  

> learning?

>

>

>

> If so, then to what extent is it (or can it be) practiced? What  

> does it look like? How does accountability for professional  

> development work? After all, professional development is an  

> important party at the table of adult literacy education.

>

>

>

> I look forward to learning your thoughts this week.

>

>

>

> Best, Jackie

>

>

>

> Jackie Taylor, Adult Literacy Professional Development List  

> Moderator, jataylor at utk.edu

>

>

>

>

>

>

>

>

>

>

>

> ----------------------------------------------------

> National Institute for Literacy

> Adult Literacy Professional Development mailing list

> professionaldevelopment at nifl.gov

>

> To unsubscribe or change your subscription settings, please go to  

> http://www.nifl.gov/mailman/listinfo/professionaldevelopment

>

> Email delivered to djrosen at comcast.net

>

> Professional Development section of the Adult Literacy Education Wiki

> http://wiki.literacytent.org/index.php/ 

> Adult_Literacy_Professional_Development


David J. Rosen
djrosen at comcast.net



----------------------------------------------------
National Institute for Literacy
Adult Literacy Professional Development mailing list
professionaldevelopment at nifl.gov
 
To unsubscribe or change your subscription settings, please go to 
http://www.nifl.gov/mailman/listinfo/professionaldevelopment

Email delivered to robinschwarz1 at aol.com 
 
Professional Development section of the Adult Literacy Education Wiki 
http://wiki.literacytent.org/index.php/Adult_Literacy_Professional_Development



 


________________________________________________________________________
More new features than ever.  Check out the new AOL Mail ! - http://webmail.aol.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.nifl.gov/pipermail/professionaldevelopment/attachments/20080107/a469fdff/attachment.html