Big Wheel Keeps on Turning

This just in, from Frank Brockmann, Center Point empresario and fellow quality crusader:

President Barack Obama announced last week that 10 states will be exempt from the requirements of the highly-criticized No Child Left Behind legislation. In exchange, those states will have to agree to a series of reforms. But some experts say the law should be scrapped completely for models that don't rely on standardized tests.

Interesting, because last night I was reading about the rise and fall of the Roman Empire California Learning Assessment System. CLAS was not the first--and surely will not be the last--testing program to be felled by public controversy. Among the many objections to CLAS were to the reading passages (two of Alice Walker's stories were excised after complaints that "Am I Blue" promoted vegetarianism, and "Roselily"was anti-religion), to the writing prompts (which were considered invasive), and to the test questions themselves (too subjective and based in emotion, according to critics).

When I began my career in hand-scoring, CLAS was one of my first projects, so I did at one time have more than a passing familiarity with the tests and how students responded to them. In general, I liked the questions (what did I know? I hadn't any content development experience at that point), although scoring did present a problem.

In some parts of the test, students were encouraged to take notes--which were called "marginalia"-- and for some questions, were offered various options as to how to respond: they could draw a picture, for example. Anyone who's ever been presented with a drawing by a small child understands the obstacle in scoring there:

You: Oh, what a lovely dinosaur.
Child: It's not a dinosaur.
You: No?
Child: No.
[pause]
You: What is it, love?
Child: It's a manatee.
You: Oh, yes, of course it is.

I see what the developers and supporters of CLAS were attempting to accomplish, and the goal is a laudable one: to lure students into engaging with authentic literature and then to welcome their genuine, individual responses to what they read. However, in retrospect, these goals might be more readily achieved at the classroom level, through both instruction and formative assessment (and by "formative assessment," I really mean all those teacherly techniques for paying attention to one's students, not the administration of a series of badly-written multiple-choice quizzes), than through a standardized testing program.

The point is really to define clearly one's purpose, and then create the assessment that will best serve that purpose, as "the man without a purpose is like a ship without a rudder," in the immortal words of Thomas Carlyle.

Big wheel keeps on turning. We went from bubble-in to performance assessment, back to bubble-in, and now we're talking about performance assessment as if it were the hot new thing, never before attempted.

Not that we shouldn't move in that direction--I like that direction very much--but if we do, there should be a moment of planning and taking stock (For which of you, intending to build a tower, sitteth not down first and counteth the cost, whether he have sufficient to finish it?), a gathering of the elders for the harvest of wise counsel, before there is a marshaling of forces. What is our purpose? What will we gain, what will we lose, is the loss worth the gain? How do we prepare teachers for this role?

What do you think?

Stakes

Today I got a shout-out asking about a practice to be employed in a classroom assessment, and it got me thinking about classroom assessments.

In this business, we think of classroom assessments as informal and low- to no-stakes, meaning that there will be no decisions made about student promotion/retention, and that there is no teacher/school/district accountability. There are probably some stakes for the students--they will receive a grade, which will contribute to an overall grade in the class, but the effect of this one grade on this one assessment is fairly minor in the great grand scheme of the universe.

Which is an excellent thing, as so few teachers know much about assessment, what skews an assessment, what practices should be avoided in assessment, which question constructions put up unnecessary obstacles for the test-takers, what kind of wording should be used. How do that happen, I wonder. Not to blame the teachers; the mystery is why teachers aren't generally required to take a class on test content development. Creating a sound assessment that accurately measures the targeted skills or knowledge requires knowing how to do it. You don't just get up from the couch one day and say, Hey, Ima build me a house without doing at least a tiny bit of research. But we don't know what we don't know.

Test content development is not difficult, but one does have to know what are the best practices in order to build a test that has at least some potential of giving a relatively fair and accurate measure of what a student knows and can do. Even if it is just in the classroom, even if the results only affect a tenth of the student's semester grade in one class.

Same thing at the DMV, you know. And those high-stakes tests could not possibly be legally defensible, constructed as they are in such a haphazard way, with all their outliers and overly attractive distractors and so on. Every single assessment professional who takes a DMV test cannot help but exclaim at the shoddy construction. What's so interesting to me about the DMV tests, though, is that even civilians recognize how bad the tests are, even if they cannot identify the reasons.