When Everything Goes South

Bad news in Florida.

Preliminary results released Monday indicate that just 27 percent of fourth-graders earned a passing score of 4.0 or better (out of 6) on the writing test. A year ago, 81 percent scored 4.0 or better. . . .

Passing scores plummeted from 81 percent to 27 percent for fourth-graders and showed similar drops in eighth and 10th grades, according to statewide results released by the Department of Education.

(Aforementioned preliminary results here.)


I did what I always do when I come across an item of note, whether it be a dime on the sidewalk or a pineapple in the headlines: I called a friend.


We compared our reactions, which were predictably (and comically, as we spoke simultaneously and in the same lexicon--not only are we friends of many years' standing, but we worked together for years, and are in the habit of oft discussing our work, a habit all the more agreeable as we share so many opinions) identical:

1. What up with the scoring? 

and/or 

2. What up with the test construction? 

and/or 

3. What up with the cohort?

and/or

 4. What up with the cut scores? 

Read more here: http://www.miamiherald.com/2012/05/14/2799146/fcat-writing-scores-plummet.html#storylink=cpy

Them's being the usual suspects. Occam's Razor.


As it happens, there was a change in the cut scores recently. The state DOE is (reportedly--I didn't talk to anyone myself) taking the line "that the results of prior years were artificially high and these are the real ones." Although the state did turn around and decide to lower the bar so more students would pass.


Take a look at the exemplar writing sets. These show examples of student essays at each score point level.


And then there may be other factors beyond the cut score. Never dismiss the possibility that someone, somewhere, made a mistake. It happens.


According to Stuart R. Kahl, Ph.d., in a paper for Measured Progress,

A test score estimates something--a student's mathematical proficiency, perhaps. It is an estimate because it is based on a small sampling of the universe of items that could have been included on the test. Further, a test score is affected by factors other than the student’s mathematical proficiency, such as: how well or motivated the student feels, whether there were distractions or interruptions during the testing session, and whether the student made good or bad guesses, to name a few. These factors, which can all be sources of measurement error, explain the difference between a student’s calculated score on a particular test and that student’s hypothetical “true” score. That true score, forever unknown, would reflect the student’s real level of proficiency.

Estimate, inference--these are the words we must use when we talk about our suppositions of what a student knows or is able to do when those suppositions are based on the results of a test.


Florida may sit herself down to rest on the lowered bar. Or there may be an investigation. The investigation will crawl through the maze of scoring operations.


If nothing turns up, the investigation could lumber over to focus on the construction of the test. Again, if the test construction is immaculate, there may be something going on with the cohort.


Or maybe it really was just the cut scores. ("Just." As if that's nothing.)


In other news: the latest on the pineapple here, along with newly reported mistakes on the New York State math tests. There's no glee in reporting that. We all of us get tarred by that brush, even if we have nothing to do with that particular test.


That's what's going on today. Who knows what tomorrow will bring.

Read more here: http://www.miamiherald.com/2012/05/14/2799146/fcat-writing-scores-plummet.html#storylink=cpy

The Pineapple in the Room

Thanks to Carmen at Anonymous Testing Company, Bob Debris (photographer extraordinaire and collector of all manner of news, from ridiculous to sublime), and a friend who'd prefer to remain anonymous and who works at Great Big Huge Test Publishing Company, I've been apprised of The Pineapple Debacle.

[Thank you, Wikipedia, for the stunning portrait of a pineapple in its native habitat.]


Author Daniel Pinkwater distances himself by saying he took the money for reprint permissions, but his original story had been edited out of all recognition. (In Pinkwater's original story, the pineapple was an eggplant, and the moral made some kind of crazy sense: "Never bet on an eggplant.")


The new moral, though ridiculous on its face, actually circles back to the suspicion of the animals that the pineapple's confidence must have been due to some trick it had up its sleeves: "Pineapples don't have sleeves."


Let me take a detour. I've been reading With Rigor for All, Second Edition: Meeting Common Core Standards for Literature by Carol Jago. (Recommended by my colleague, publisher, and partner-in-crime comrade-for-quality-reform Frank Brockmann.) We've discussed rigor here previously. We all of us--educators, assessment content developers, curriculum experts, writers of children's literature--should read this book.


Carol Jago talks about the need for stories and how genuinely responding to what we read not only enriches our lives but serves a need all we humans have for narrative. What is life, if not a story we live? With the twists and turns and reversals and gifts of the unexpected? Where the mighty are humbled, the lowly are risen up, where we commit follies, but where we find redemption? Where we rejoice with those who rejoice, and mourn with those who mourn?


In stories, we connect to what lay hidden in ourselves; we connect to each other; we connect to every human who has ever lived. In stories, we see ourselves as we are--and as we wish to be. We wrestle with our demons and angels. This is why the study of literature is so important. Our character is molded from the stories we tell ourselves.


And yet, we don't have faith in our children. We don't have faith that they will understand the complicated beautiful mess of being human. We don't have faith they will understand the complex beautiful machinery of language. We fail to offer them stories to which they could genuinely respond. Instead, we offer passages that insult their intelligence. 


Hence the story about a talking pineapple. Offered with good intentions, you understand (similar good intentions run amok discussed here). The good intentions went something like this: Those poor kids, might as well give them a story that will make them laugh.


And hence my book. Due out in June, God willing and the creek don't rise. Our launch of the first offensive maneuver in the campaign for quality reform in assessment content development: we'll elevate the quality by recruiting real writers to write for reading tests. Our experiments so far have been promising.


As far as what happened with Pinkwater's passage--quite frankly, I'm mystified. Many of my stories, poems, and articles have been reprinted in high-stakes assessments. None have been edited to that degree. (Once, a company asked if they could change the gender of the main character, as they felt their test was out of balance in that regard.) That is a new one on me. Most authors will not allow that kind of editing. When I myself worked at Great Big Huge Test Publishing Company, from associate editor to development manager, I would not have so edited a passage. But the real problem wasn't the editing; it was the selection of the passage itself. Absurdity, though fun to read, is better for classroom instruction than for assessment.


Because humor is relative. As is driven home to me when I realize that the person who laughs most at my jokes is me. (Or I, if you be a stickler for grammar.)


UPDATE: I just thought of a possible reason for the sleeveless pineapple. The test blueprint may have included a standard targeting the use of idiom, and so the editor decided the easiest solution was to edit the passage to add the "up its sleeve" idiom.


As for the transformation from eggplant to pineapple, I remain mystified. Maybe someone on one of the review committees hated eggplant. Sounds like a joke, but I'm not kidding. Once I was in a passage review meeting at which the client picked up a story, looked at it, wrinkled her nose, and threw it on the floor. "I hate cats," she said.




UPDATE THE SECOND: Removed a link and anonymized an identity.