What the World Needs Now

It may be unnecessary to say that in talking about what's wrong, I don't mean to pick on anybody in particular. Even if it is one person making specific mistakes, I tend to see this as a systemic ill, as previously discussed, rather than an opportunity to dogpile on an unwitting offender.

A systemic ill that is particularly pernicious in the test publishing industry. How can we expect kids to do well on tests when the tests themselves are riddled with error?

The safeguards are so simple:

1. provide adequate training (again, as previously discussed)

2. follow some kind of standard process for detecting and repairing error

3. use the information from #2 to supplement #1

These safeguards cost time and money, which explains why so many companies have gradually let them fall by the wayside. 

Training takes time and attention. I can't believe how fortunate I was to have received the training I did. In my first year as an associate editor, I probably had more training opportunities than editors now receive in a decade. It was a different era. The Great Big Huge Test Publishing Company at which I worked had an army of style editors, proofreaders, and QA personnel.

Over the last dozen years, I've seen companies completely dismantle their style editing departments and outsource the work to freelance editors. Now style editors (copy editors) are held in such low regard that if any Bright Young Thing with a talent for grammar and language conventions tells me she'd like to give this work a try, I steer her away from style editing and into content.  I hate doing it. I myself hold style editors in high esteem, maybe because often their job seems to be to keep me from making a fool of myself. What the world needs now--no disrespect intended to Dionne Warwick and Burt Bacharach--is more style editing.

Many companies have also dismantled their content development departments.

So there's no real continuity for item writers and content editors. They work on a project for one company and then skip to a project for another. The content development people who remain at the companies are overwhelmed and certainly don't have the time to provide training to freelancers; editors accept or reject items, which means they're likely to fix mistakes themselves, and so the item writer never hears of his mistakes and is free to repeat those mistakes forever in a state of unfortunate ignorant complacence.

Right now, I'm thinking of all this in terms of language conventions items, because I was reviewing a set this morning, but I'm sure this applies to all domains and content areas.

Right now, I offer some specific observations about language items:
1. When language items accompany an editing passage that contains embedded error, the content of each item must be mutually exclusive. A sentence from the passage should contain only one error, and should be used only for one item in a set.
2. "Syntax" has to do with grammatical rules; "diction" has to do with the writer's choice of words.
3. An item should clearly target only one type of error, and should not mix types of errors. For example, a punctuation item shouldn't include distractors that contain errors in verb tense.

More previously discussed here.

As an aside, it's interesting how you can learn a lot about a person by reviewing his or her work, and how this body of knowledge grows over time. It's like how my mechanic told me that he knows more about his customers from the condition of their cars than he does from talking with them. He can tell who drives erratically, who slams on the brakes, who grinds the gears, and he can tell who doesn't. Everything is everything. You can develop a similar personality profile simply by reviewing a writer's work.

I can't tell you how many times I've finished working on a manuscript, items or reading passage, don't make no difference, and felt a deep and tender affection for the writer because his or her work demonstrated care, attention, and conscientiousness. 

UPDATE: Forgot to mention. I started writing about all this because I saw this book mentioned in the Common Core Standards and immediately ordered it. 

When Everything Goes South

Bad news in Florida.

Preliminary results released Monday indicate that just 27 percent of fourth-graders earned a passing score of 4.0 or better (out of 6) on the writing test. A year ago, 81 percent scored 4.0 or better. . . .

Passing scores plummeted from 81 percent to 27 percent for fourth-graders and showed similar drops in eighth and 10th grades, according to statewide results released by the Department of Education.

(Aforementioned preliminary results here.)


I did what I always do when I come across an item of note, whether it be a dime on the sidewalk or a pineapple in the headlines: I called a friend.


We compared our reactions, which were predictably (and comically, as we spoke simultaneously and in the same lexicon--not only are we friends of many years' standing, but we worked together for years, and are in the habit of oft discussing our work, a habit all the more agreeable as we share so many opinions) identical:

1. What up with the scoring? 

and/or 

2. What up with the test construction? 

and/or 

3. What up with the cohort?

and/or

 4. What up with the cut scores? 

Read more here: http://www.miamiherald.com/2012/05/14/2799146/fcat-writing-scores-plummet.html#storylink=cpy

Them's being the usual suspects. Occam's Razor.


As it happens, there was a change in the cut scores recently. The state DOE is (reportedly--I didn't talk to anyone myself) taking the line "that the results of prior years were artificially high and these are the real ones." Although the state did turn around and decide to lower the bar so more students would pass.


Take a look at the exemplar writing sets. These show examples of student essays at each score point level.


And then there may be other factors beyond the cut score. Never dismiss the possibility that someone, somewhere, made a mistake. It happens.


According to Stuart R. Kahl, Ph.d., in a paper for Measured Progress,

A test score estimates something--a student's mathematical proficiency, perhaps. It is an estimate because it is based on a small sampling of the universe of items that could have been included on the test. Further, a test score is affected by factors other than the student’s mathematical proficiency, such as: how well or motivated the student feels, whether there were distractions or interruptions during the testing session, and whether the student made good or bad guesses, to name a few. These factors, which can all be sources of measurement error, explain the difference between a student’s calculated score on a particular test and that student’s hypothetical “true” score. That true score, forever unknown, would reflect the student’s real level of proficiency.

Estimate, inference--these are the words we must use when we talk about our suppositions of what a student knows or is able to do when those suppositions are based on the results of a test.


Florida may sit herself down to rest on the lowered bar. Or there may be an investigation. The investigation will crawl through the maze of scoring operations.


If nothing turns up, the investigation could lumber over to focus on the construction of the test. Again, if the test construction is immaculate, there may be something going on with the cohort.


Or maybe it really was just the cut scores. ("Just." As if that's nothing.)


In other news: the latest on the pineapple here, along with newly reported mistakes on the New York State math tests. There's no glee in reporting that. We all of us get tarred by that brush, even if we have nothing to do with that particular test.


That's what's going on today. Who knows what tomorrow will bring.

Read more here: http://www.miamiherald.com/2012/05/14/2799146/fcat-writing-scores-plummet.html#storylink=cpy

The Problem with Pink Celluloid

You may know, as I did not until today, that celluloid was the first thermoplastic. It was first made in the mid 1800s, and was a cheap replacement for ivory. It's very flammable, though, and is quick to decompose, and so fell out of favor, and is now mainly used in guitar picks and ping pong balls. Who knew. [Thank you, Wikipedia.]


A few days ago, I was talking with a middle school teacher, the kind of middle school teacher you wish you had had, a teacher who is young and alive enough that students might be able to imagine that she, too, was once in 7th grade.


She was telling me of a district writing prompt accompanied by a reading passage, the understanding of which depended on decoding the phrase "pink celluloid dresser set."


Perhaps because I spent my entire adolescence reading, I did know what this meant. Then again, I have been reading for many, many years now. (Mainly Victorian novels.) I think maybe one 7th grader in that district did. I asked my 8th grade daughters what they thought a pink celluloid dresser set would be, and they guessed furniture.


So all these 7th graders read this story with the pink celluloid dresser set, and then were given these directions:

Write an essay in which you present your understanding of the characters and the overall meaning of the story. Support your ideas with examples and/or evidence from the text. 

Remember that your writing will be scored on how well you write an essay that:

• shows your understanding of the author's message and your insight into the characters and ideas presented in the story;
• is organized around several clear ideas and/or images from the story;
• justifies your interpretation by giving examples and citing evidence from the text; and 

• uses correct grammar, spelling, punctuation, and capitalization.

The problem with the story could easily have been solved with a footnote explaining that  celluloid is a kind of plastic, and a dresser set is a collection that includes a comb, hairbrush, and hand mirror.


The problem with the prompt is a bit bigger.


What should be the focus of the essay? It's difficult to tell.


At first, you think the students' understanding of the characters and the overall meaning is the focus, which is not really that clear of a target, especially considering the confusion caused by the pink celluloid dresser set. (And I think we should be able to assume that 7th graders know what theme is, and dispense with "the overall meaning.")


But okay, the students are going to write about the characters, probably Katrin and Mama, although they might also write about Papa, Nels, Christine, Dagmar, and Mr. Schiller. And they are going to write about the theme, which has something to do with generosity and sacrifice (it's a redo of "The Gift of the Magi" by O. Henry). 


What exactly about the characters are they supposed to write? What exactly about the theme? The bulleted list is no help, because it introduces the need to write about the author's message and to organize the writing "around clear ideas and images from the story."


If I were a 7th grader, I wouldn't know what to write. If it's their understanding, many of the essays will probably start with "I didn't really understand the story."


This prompt couldn't elicit great writing. That's too bad, because maybe some of these 7th graders are great writers.


We could easily rewrite the prompt:

Write an essay in which you describe how Katrin changes from the beginning to the end of the story. 

In the essay, be sure to:

• describe how Katrin changes in the story

• explain why Katrin changes

• support your explanation with specific details

• use correct grammar, spelling, and punctuation

Or, if we want to focus on the symbolism of the pink celluloid dresser set and the brooch:

Write an essay in which you explain what the pink celluloid dresser set and the brooch represent in the story.

In the essay, be sure to:

• explain the meaning of the dresser set and the brooch

• support your explanation with specific details

• use correct grammar, spelling, and punctuation

Or we could focus on theme, or Mama's sacrifice, or Christine's role in the story, or how Katrin's covetousness of the pink celluloid dresser set caused her to be blind to all common sense and family feeling, but her sacrifice transformed her.


There is plenty to write about. It's not a bad story; it is old-fashioned, which is a little unfortunate in a mandatory assessment, because that alone will distance a lot of kids.

Then again, I asked one of my daughters to read the prompt. She eyed it suspiciously.

Second Daughter: Is that a response to literature?
Me: What do you think?
Second Daughter: It's a response to literature.
Me: What are you supposed to write?
Second Daughter: The first paragraph is an introduction. In the second paragraph, you write about the theme. Then you write about the characters and give examples. Then you write a conclusion.
Me: Did you understand that prompt?
Second Daughter: I have that prompt memorized. That prompt is all we did in 7th grade.

She hadn't even seen the words "response to literature" on the previous page when she read the prompt.


As an assessment content developer, I want better for the students. As a parent, I want better for my children. Starting with opportunities to genuinely respond to literature.