Subscribers, the following post is from Ajit Gopalakrishnan.

------------------------------------------------------------------------
----

Please pardon this lengthy post.

Standardized achievement assessments that measure overall literacy
ability (e.g. CASAS, TABE, NAAL, NAEP, GED, etc.), especially if they
are group-administered and involve selected response items, may not
provide the instructor with an exhaustive and in-depth analysis of every
facet of the basic skill for every learner in the classroom. Good
teachers and programs routinely supplement with formative assessments.
That said, results from standardized achievement assessments can inform
the teacher of the ability level for each student. An item analysis can
also tell the teacher about the general areas of strengths and
weaknesses for individual students and the class as a whole --
information that can inform curriculum/lesson planning.

In the CASAS system (which I am most familiar with), the CASAS
competencies form the curricular backbone. These competencies were
developed and are periodically revalidated with a broad stakeholder
constituency. In recent years, content standards have also been
developed that identify the basic skills that underlie these
competencies. Priority competencies identified by these stakeholders
form the basis for assessment development.

These priority competencies act like "power standards". However, when
developing curriculum and lesson plans, we recommend that the
instructional content go beyond the competencies measured on the
pre-test and incorporate other competencies from the master list that
may also be important and relevant to the learners. Because we are
talking about measuring basic skill ability and not subject matter
knowledge like social studies or biology, instruction in a broader set
of competencies and the basic skills that underlie them can facilitate
transferable learning that ultimately contributes to greater performance
on the post-assessment.

When it comes to measuring progress, I feel that a CASAS
post-test-assessment does measure what students are learning because it
is assessing learners on the same standards (i.e. CASAS competencies and
basic skill content standards) and performance levels that were used for
(i) the baseline pre-test assessment, (ii) the development of
curriculum, and (iii) the delivery of instruction.

Because the NRS expects learners to substantially improve their literacy
ability from pre-to-post test by demonstrating movement from one
literacy level to the next, progress in one or two sub-components of a
basic skill (e.g. converting percents to fractions or learning a new
grammar rule) may in and of itself be insufficient to achieve an NRS
level completion. This can be frustrating for a teacher who may have
noticed student progress in class work but not see it reflected in the
standardized assessment results.

Sometimes, even if the curriculum and instruction are aligned to the
standards, learners may get tripped up in the assessment by a certain
vocabulary word, by the context of a test item (Jim gave the example of
his student who was Buddhist monk from Cambodia), one of the
distractors, a writing prompt, etc. Additionally, though standardization
brings validity and reliability to the process of measuring progress
(which I believe has been vital to maintaining the credibility and
funding of adult education at all levels), it [standardization] tags
along some "standard measurement error" wherein some fluctuation in
scores is not out of the realm of accepted possibility. Computer
adaptive testing can minimize this measurement error even further but it
may lessen the diagnostic information provided because different
students are administered different test items. All these issues are not
unique to assessments used for the NRS but are inherent to the process
of standardization.

Results from assessments that require students to "construct" their
response instead of "selecting" from a set of choices can help this
matter somewhat by providing more information that informs instruction.
Over the past 10 years, CASAS has developed standardized constructed
response assessments such as the functional writing assessment and the
workplace speaking assessment. Here, the student responses provide much
richer information to inform instruction than might be available from a
multiple choice test. With extensive training, ongoing recertification,
and stringent inter-rater reliability practices, these tests can also
serve as reliable standardized assessments for NRS accountability
purposes. The challenges here are related to cost and scalability. These
assessments are administered and/or scored one-on-one and the adult
education system is just not resourced currently to implement this level
of assessment for all learners in the system.

Despite all these issues, in my experience, programs that reflect good
outcomes on standardized assessments (both constructed and selected
response) encompass more of the characteristics of high quality
programs, and are more effective at helping learners to achieve their
goals. They have strong leaders, aligned curriculum and instructional
practices throughout the program, instruction of reasonable intensity
and duration, counseling and other support services, and higher rates of
learner persistence and attendance. This has been reaffirmed for me
through extensive analyses of Connecticut's performance and funding
data, reviews of local curriculum, observations of classroom
instruction, and conversations with practitioners in CT and in other
states.

We are also beginning to see in our data that achieving higher levels of
basic skill proficiency as evidenced on CASAS assessments translates to
measurable success outside the classroom e.g. greater probabilities of
passing the GED test, greater employment rates, greater average
earnings, etc.

One thing I will say about the NRS reporting framework is that instead
of using a level completion approach, a growth model may more accurately
capture the progress a student is making, regardless of his/her starting
point in a functioning level. For example, in one of our high performing
programs, only 18% of learners at the advanced ESL level (which spans 15
CASAS scale score points) completed that level while 58% of learners
demonstrated a significant gain of 4-scaled score points from
pre-to-post test. For this reason, in CT, we evaluate scaled-score point
growth as a separate measure within our accountability framework that is
different from level completion.

In conclusion, the big challenge from my perspective is not that the
assessments do not reliably capture the progress made by learners or
measure what is taught in the classroom. The real challenge is that a
majority of the ESL students just don't stay long enough to make
significant progress, regardless of the assessment used. As I stated
earlier, two CASAS studies with CA and CT data and now the recent BEST
study with MA and IL data confirm that higher success rates are
attainable if students stay for at least 100 hours in a fiscal year, but
the reality is that a majority of ESL students don't do that. One just
needs to compare the results from NRS Table 4 (all learners) with that
of Table 4B (those with matched pre-post tests), for any state that has
sound assessment practices and a reliable data system, to see this
persistence issue manifested over and over and over again. Strategies to
improve persistence - now that's another discussion.

Ajit


Ajit Gopalakrishnan
Connecticut Department of Education
25 Industrial Park Road
Middletown, CT 06457
Phone: (860) 807-2125
Fax: (860) 807-2062
Email: ajit.gopalakrishnan at ct.gov





-----Original Message-----

From: assessment-bounces at nifl.gov [mailto:assessment-bounces at nifl.gov]
On Behalf Of Schneider, Jim

Sent: Friday, February 08, 2008 12:30 PM

To: The Assessment Discussion List

Subject: [Assessment 1190] Re: {Dangerous Content?} assessments



Forrest,



Having read everything I can find that you have written in the past 17

years, dim is the least of terms I'd use to describe you. (I'm working

on a Ph.D proposal dealing with the marginality of community college ABE

programs).



The issue with standardized tests, NRS, NCLB, etc. is that they are

trying to impose Frederick Taylor's principles of scientific management

to people. The variables involved with forging steel are certainly

complex, but finite in comparison to the variables involved with nearly

any aspect of working with people.



An incredible instructor can make a poorly conceived curriculum work,

whereas the best curriculum in the world will languish in the hands of a

mediocre to poor teacher.



Similarly, standardized assessments are a weak, artificial means of

assessing skills.



I do like CASAS for the life & work orientation over a more academic

assessment such as TABE. However, even the life and work orientation is

highly dependent on the experiences of the learner. This morning I have

a Buddhist monk from Cambodia enrolling in our program. He has completed

16 years of education in Cambodia, his listening, speaking, and writing

are all exemplary relative to our typical ESL student. However, he is

struggling with the last section on a CASAS C level reading assessment

because of the context of the material being presented. I suspect that

he will do well, but his struggle exemplifies the weakness of using

standardized assessment and expecting standardized results. Educating

people is significantly more complex than forging steel... We are

learning how to become more effective in our approaches, but I cannot

fathom teaching and learning ever being "standardized" to the extent of

those who seem to believe that NRS/NCLB etc. is the answer to the

question.



Jim S.



-----Original Message-----

From: assessment-bounces at nifl.gov [mailto:assessment-bounces at nifl.gov]

On Behalf Of Forrest Chisman

Sent: Friday, February 08, 2008 10:14 AM

To: 'The Assessment Discussion List'

Subject: [Assessment 1183] Re: {Dangerous Content?} assessments



Jim,



Very interesting, and thanks. What puzzles me is that a lot of people

seems to say, as you do, that CASAS and other standardized tests work

well for NRS, but not for teaching/learning. But if the purpose of NRS

is to determine how much students are learning in particular programs,

how can this be so? Maybe I'm just dumb about this, but I'd rather

confess my ignorance than live with it. Shouldn't NRS be measuring what

your students are actually learning? And if CASAS and other standardized

tests don't do that, don't we need either a better assessment system, or

a better way of reporting to NRS by other means.



As I say, maybe I'm just dim.



Forrest      



-----Original Message-----

From: assessment-bounces at nifl.gov [mailto:assessment-bounces at nifl.gov]

On Behalf Of Schneider, Jim

Sent: Friday, February 08, 2008 10:31 AM

To: The Assessment Discussion List

Subject: [Assessment 1177] Re: {Dangerous Content?} assessments



CASAS is the assessment of choice (mandated) in Iowa. If teachers and

learners buy into competency-based instruction, the CASAS assessments

have some utility. The short-comings and artificiality of standardized

tests seem to be amplified with the ESL.



The reading and listening tests fall short of providing a comprehensive

useful assessment of the student's abilities at orientation. We offer

three levels in the morning, beginning, intermediate, and advanced, with

the placement being done using their writing (application), speaking and

listening (initial interview questions), and reading (based on the

literacy demonstrated with the application and interview).



However, due to the mix of skills, the beginning and intermediate are

really blended multi-level classes with some beginners who can read, but

struggle with speech and writing, and some intermediate who can speak

very clearly, but struggle with the reading and writing and virtually

every other possible combination.



Similarly, teachers and students have been frustrated with the lack of

progress in post-testing as well as the inconsistency of the scores.

Those students that bring high literacy skills to the class tend to have

fairly consistent scores. Those who are not as literate (and are a

growing population in ESL) can swing up and down the scaled score

throughout the year with little rhyme or reason.



Bottom line - CASAS works well for the standardization required by NRS,

but is not as powerful or useful in regard to teaching and learning as

one would like.



Jim Schneider



-----Original Message-----

From: assessment-bounces at nifl.gov [mailto:assessment-bounces at nifl.gov]

On Behalf Of Forrest Chisman

Sent: Thursday, February 07, 2008 8:16 PM

To: 'The Assessment Discussion List'

Subject: [Assessment 1170] Re: {Dangerous Content?} assessments



Gail,



Good question!!!!!!!!!! I'd be interested in the replies. Because CASAS

is MANDATED in many states, and most programs don't want to use more

than one assessment (because of limited time for testing and fear of

over-testing).

What do you all think of CASAS for ESL?



Forrest



-----Original Message-----

From: assessment-bounces at nifl.gov [mailto:assessment-bounces at nifl.gov]

On Behalf Of Gail Burnett

Sent: Thursday, February 07, 2008 7:08 PM

To: assessment at nifl.gov

Subject: [Assessment 1162] {Dangerous Content?} assessments



Warning: This message has had one or more attachments removed

Warning: (not named).

Warning: Please read the "AttachmentWarning.txt" attachment(s) for more

information.



All the discussion on this list (the first I've ever joined) has been

fascinating. I have a question I haven't seen addressed. Someone said

that the BEST-Plus is the only valid kind of test for ESL learners. What

does anyone think of the CASAS? That's what we use with our adult

students, both ELLs and native speakers. I know it wasn't designed for

ESL use but it does use real-life material (road maps, pay stubs, store

signs, etc.) and I think that's good. What has always bothered me is

that it's taken silently. That doesn't seem right for our lower level

students, who rarely make many gains.

I wonder if it's considered valid for them or not. I know that other ESL

programs do use it.



Thanks.