Saturday, April 28, 2012

Pass the Pineapple

This, from Jo Perry, the beginning of a discussion about the larger context for the sleeveless talking pineapple:

An American child could go to a public school run by Pearson, studying from books produced by Pearson, while his or her progress is evaluated by Pearson standardized tests. The only public participant in the show would be the taxpayer.
If all else fails, the kid could always drop out and try to get a diploma via the good old G.E.D. The General Educational Development test program used to be operated by the nonprofit American Council on Education, but last year the Council and Pearson announced that they were going into a partnership to redevelop the G.E.D. — a nationally used near-monopoly — as a profit-making enterprise.

I'm very interested in this conversation. I'll say upfront that although I find it disagreeable to point at problems without offering possible solutions, this one's got me baffled.

There are not-for-profits that publish curriculum and assessment materials. From what I've seen, many operate just as corporations do, but perhaps more cheerfully, said operations being subsidized by what I imagine are tax breaks that lend some comfort to the proceedings.

Twice I've been recruited by not-for-profit agencies that publish test materials. Nothing seemed any different than any corporation. During the come-work-with-us talk, the VP assured me that just because their agency was a not-for-profit, this did not mean they didn't make a profit. He told me they liked to think of themselves as a meritocracy, and then he wrote a salary figure on a piece of paper and slipped it across the table.

From what I've seen of public education--and I've spent a tremendous time in classrooms at every grade, in review meetings with teachers, administrators, and other education professionals, and in state DOE conference rooms--I can't say that the public sector manages anything better than businesses or not-for-profits do.

(The elephantine factor is one problem. The larger a system is, the more difficult its management.)

In the immortal words of Tolstoy, "Everyone thinks of changing the world, but no one thinks of changing himself."*

Every time I've emerged from a classroom or a conference room (or even a presentation at an industry conference) feeling optimistic, it's been because of one person. A person who cares and whose work and words show that she cares. (I use "she" out of habit, not to be exclusionary.) There are brilliant and dedicated teachers in our schools. There are brilliant and dedicated leaders in education. (Some of these work with the corporations, by the way--there are certain names that always reassure me even before I read the recommendations based on their research.) There are people working in the corporations who are deeply and sincerely dedicated to serving students in their work.

There are many who aren't.

My feeling is that whatever your work, if you're just in it for the paycheck, you're doing yourself, your employer, and the end-user a tremendous disservice. We humans need to find and serve a higher meaning.

It shows when we don't. It shows, whether we work behind the counter at Starbucks or with a bunch of tiny little savages kindergarteners in an elementary school, or in a partitioned cubicle in a big corporation.

* I hope you understand I don't mean Gail Collins when I say this. She is calling our attention to a matter worthy of discussion.

Wednesday, April 25, 2012

Start Where You Are

The first time my second daughter began to read To Kill a Mockingbird, she gave up within ten pages. She was in the fifth grade. The reading was so difficult that she got no pleasure from it. When she read it last year, in the 7th grade, she loved it.

When we started the homeschooling, my first daughter was reading The Great Gatsby and my second daughter was reading Rebecca. (Both were thrilled with their choices, but neither had much interest in the other's.) When they finished, I was thinking about what to suggest next. I wanted them to read the same book so our literature class would be more focused than it had been. I'd told a friend that my first daughter had loved a YA historical novel set at turn of the century, and my friend said why not Edith Wharton.

The plot of The Buccaneers seems perfect for 14-year-olds: a coterie of friends of differing temperaments and sensibilities poised at the brink of making life's big decisions.

But my daughters wanted to start reading right away, and we couldn't find The Buccaneers at any of the local bookstores, neither chain nor independent nor used. Not even our library had a copy. I'd loaned mine out and you know how that goes. We chose The Age of Innocence instead. 

Almost immediately, my second daughter said it was too hard. My first daughter agreed it was hard, but was willing to persevere. For two weeks, my second daughter lagged behind in her reading. Then, realizing her sister had left her in the dust, she buckled down to the task. This was two days ago.

As I was making dinner tonight, she showed me how few pages she had left to read (that would be three). In two days, she'd read more than 180 pages.
Me: What happened?
Second Daughter: I started liking it, and then I liked it so much I couldn't stop.
This happened to me with Moby Dick, though I was a much later bloomer. I'd tried to read it many times, from high school onward, but wasn't able to get past the first chapter (which is very unusual for me, I hate quitting a book, it just feels wrong) until I was nearly 30 and in grad school. Why? I have no idea. Maybe I was immature. Maybe it is simply that I would have loved it if I had persisted.

These matters sound little, but they are the matters that make up reading. What do we do when the text is just too difficult? How can we tell when the difficulty may be overcome once the reader is engaged, or whether the student needs to develop the muscles for the heavy lifting? If the latter, what's the best way to nudge the student into gaining skills and yet not push so hard that the student becomes discouraged?

I've talked previously about my remedial community college students, how some were surpassed by 4th graders when it came to writing skills. Ditto reading. It was a challenge. 

I tried to go at the problem in different ways. We read a lot in class. I assigned an anthology of short short fiction (Sudden Fiction) which they liked and actually did read. I often brought in copies of articles and essays from newspapers and magazines on topics that I thought they might like: "The Ways We Lie" by Stephanie Ericsson, "On Dumpster Diving" by Lars Eighner, "Starting Over with God" by Douglas Coupland, "Bet with America" by William "Upski" Wimsatt. I brought in stories by great writers who were also my friends: "Close Calls" by Josh Schneyer, "New Pants" by Jervey Tervalon, "Someone's Got Cold Feet" by Kia Penso.

Most of my students worked hard. Much of the reading was neither easy nor natural to them; Carol Jago talks about how we need to talk with students about working at reading and teach them how to persist in the face of polysyllabic words and complex syntax. Nor did we begin with what was most difficult--we started where they were.

And how should this relate to assessment? A friend (who's been in the business long enough and at enough different companies to see trends whisk in and fade away) and I were talking today about rigor and the Common Core Standards and how the standards require what many students simply aren't capable of. Yet. I certainly don't mean never. I just mean their skills need to get stronger. We wonder how states are going to address this problem.

We can't develop sophisticated rigorous tests that all but the top ten percent will fail. We can't develop simple less demanding tests that all but the lowest ten percent will pass. 

And yet we must expect more from our students in order that they develop the skills they'll need as adults, one of those skills simply being that of keeping at it even when it is hard to do.

Saturday, April 21, 2012

The Pineapple in the Room

Thanks to Carmen at Anonymous Testing Company, Bob Debris (photographer extraordinaire and collector of all manner of news, from ridiculous to sublime), and a friend who'd prefer to remain anonymous and who works at Great Big Huge Test Publishing Company, I've been apprised of The Pineapple Debacle.

[Thank you, Wikipedia, for the stunning portrait of a pineapple in its native habitat.]

Author Daniel Pinkwater distances himself by saying he took the money for reprint permissions, but his original story had been edited out of all recognition. (In Pinkwater's original story, the pineapple was an eggplant, and the moral made some kind of crazy sense: "Never bet on an eggplant.")

The new moral, though ridiculous on its face, actually circles back to the suspicion of the animals that the pineapple's confidence must have been due to some trick it had up its sleeves: "Pineapples don't have sleeves."

Let me take a detour. I've been reading With Rigor for All, Second Edition: Meeting Common Core Standards for Literature by Carol Jago. (Recommended by my colleague, publisher, and partner-in-crime comrade-for-quality-reform Frank Brockmann.) We've discussed rigor here previously. We all of us--educators, assessment content developers, curriculum experts, writers of children's literature--should read this book.

Carol Jago talks about the need for stories and how genuinely responding to what we read not only enriches our lives but serves a need all we humans have for narrative. What is life, if not a story we live? With the twists and turns and reversals and gifts of the unexpected? Where the mighty are humbled, the lowly are risen up, where we commit follies, but where we find redemption? Where we rejoice with those who rejoice, and mourn with those who mourn?

In stories, we connect to what lay hidden in ourselves; we connect to each other; we connect to every human who has ever lived. In stories, we see ourselves as we are--and as we wish to be. We wrestle with our demons and angels. This is why the study of literature is so important. Our character is molded from the stories we tell ourselves.

And yet, we don't have faith in our children. We don't have faith that they will understand the complicated beautiful mess of being human. We don't have faith they will understand the complex beautiful machinery of language. We fail to offer them stories to which they could genuinely respond. Instead, we offer passages that insult their intelligence. 

Hence the story about a talking pineapple. Offered with good intentions, you understand (similar good intentions run amok discussed here). The good intentions went something like this: Those poor kids, might as well give them a story that will make them laugh.

And hence my book. Due out in June, God willing and the creek don't rise. Our launch of the first offensive maneuver in the campaign for quality reform in assessment content development: we'll elevate the quality by recruiting real writers to write for reading tests. Our experiments so far have been promising.

As far as what happened with Pinkwater's passage--quite frankly, I'm mystified. Many of my stories, poems, and articles have been reprinted in high-stakes assessments. None have been edited to that degree. (Once, a company asked if they could change the gender of the main character, as they felt their test was out of balance in that regard.) That is a new one on me. Most authors will not allow that kind of editing. When I myself worked at Great Big Huge Test Publishing Company, from associate editor to development manager, I would not have so edited a passage. But the real problem wasn't the editing; it was the selection of the passage itself. Absurdity, though fun to read, is better for classroom instruction than for assessment.

Because humor is relative. As is driven home to me when I realize that the person who laughs most at my jokes is me. (Or I, if you be a stickler for grammar.)

UPDATE: I just thought of a possible reason for the sleeveless pineapple. The test blueprint may have included a standard targeting the use of idiom, and so the editor decided the easiest solution was to edit the passage to add the "up its sleeve" idiom.

As for the transformation from eggplant to pineapple, I remain mystified. Maybe someone on one of the review committees hated eggplant. Sounds like a joke, but I'm not kidding. Once I was in a passage review meeting at which the client picked up a story, looked at it, wrinkled her nose, and threw it on the floor. "I hate cats," she said.

UPDATE THE SECOND: Removed a link and anonymized an identity.

Friday, April 20, 2012

The Problem with Pink Celluloid

You may know, as I did not until today, that celluloid was the first thermoplastic. It was first made in the mid 1800s, and was a cheap replacement for ivory. It's very flammable, though, and is quick to decompose, and so fell out of favor, and is now mainly used in guitar picks and ping pong balls. Who knew. [Thank you, Wikipedia.]

A few days ago, I was talking with a middle school teacher, the kind of middle school teacher you wish you had had, a teacher who is young and alive enough that students might be able to imagine that she, too, was once in 7th grade.

She was telling me of a district writing prompt accompanied by a reading passage, the understanding of which depended on decoding the phrase "pink celluloid dresser set."

Perhaps because I spent my entire adolescence reading, I did know what this meant. Then again, I have been reading for many, many years now. (Mainly Victorian novels.) I think maybe one 7th grader in that district did. I asked my 8th grade daughters what they thought a pink celluloid dresser set would be, and they guessed furniture.

So all these 7th graders read this story with the pink celluloid dresser set, and then were given these directions:

Write an essay in which you present your understanding of the characters and the overall meaning of the story. Support your ideas with examples and/or evidence from the text. 
Remember that your writing will be scored on how well you write an essay that:
  • shows your understanding of the author's message and your insight into the characters and ideas presented in the story;
  • is organized around several clear ideas and/or images from the story;
  • justifies your interpretation by giving examples and citing evidence from the text; and 

  • uses correct grammar, spelling, punctuation, and capitalization.
The problem with the story could easily have been solved with a footnote explaining that  celluloid is a kind of plastic, and a dresser set is a collection that includes a comb, hairbrush, and hand mirror.

The problem with the prompt is a bit bigger.

What should be the focus of the essay? It's difficult to tell.

At first, you think the students' understanding of the characters and the overall meaning is the focus, which is not really that clear of a target, especially considering the confusion caused by the pink celluloid dresser set. (And I think we should be able to assume that 7th graders know what theme is, and dispense with "the overall meaning.")

But okay, the students are going to write about the characters, probably Katrin and Mama, although they might also write about Papa, Nels, Christine, Dagmar, and Mr. Schiller. And they are going to write about the theme, which has something to do with generosity and sacrifice (it's a redo of "The Gift of the Magi" by O. Henry). 

What exactly about the characters are they supposed to write? What exactly about the theme? The bulleted list is no help, because it introduces the need to write about the author's message and to organize the writing "around clear ideas and images from the story."

If I were a 7th grader, I wouldn't know what to write. If it's their understanding, many of the essays will probably start with "I didn't really understand the story."

This prompt couldn't elicit great writing. That's too bad, because maybe some of these 7th graders are great writers.

We could easily rewrite the prompt:

Write an essay in which you describe how Katrin changes from the beginning to the end of the story. 
In the essay, be sure to:
  • describe how Katrin changes in the story
  • explain why Katrin changes
  • support your explanation with specific details
  • use correct grammar, spelling, and punctuation
Or, if we want to focus on the symbolism of the pink celluloid dresser set and the brooch:

Write an essay in which you explain what the pink celluloid dresser set and the brooch represent in the story.
In the essay, be sure to:
  • explain the meaning of the dresser set and the brooch
  • support your explanation with specific details
  • use correct grammar, spelling, and punctuation

Or we could focus on theme, or Mama's sacrifice, or Christine's role in the story, or how Katrin's covetousness of the pink celluloid dresser set caused her to be blind to all common sense and family feeling, but her sacrifice transformed her.

There is plenty to write about. It's not a bad story; it is old-fashioned, which is a little unfortunate in a mandatory assessment, because that alone will distance a lot of kids.

Then again, I asked one of my daughters to read the prompt. She eyed it suspiciously.
Second Daughter: Is that a response to literature?
Me: What do you think?
Second Daughter: It's a response to literature.
Me: What are you supposed to write?
Second Daughter: The first paragraph is an introduction. In the second paragraph, you write about the theme. Then you write about the characters and give examples. Then you write a conclusion.
Me: Did you understand that prompt?
Second Daughter: I have that prompt memorized. That prompt is all we did in 7th grade.
She hadn't even seen the words "response to literature" on the previous page when she read the prompt.

As an assessment content developer, I want better for the students. As a parent, I want better for my children. Starting with opportunities to genuinely respond to literature.

Friday, April 13, 2012

What Hath Been, What Will Be

There's nothing new under the sun.

AI scoring is always lurking at the edges of assessment talk. Because scoring depends on human labor, and is therefore expensive. Wouldn't it be great to automate scoring, eliminate human workers, and save a ton of money?

Erik Robelen at Curriculum Matters brings up the not surprising results of a study indicating that AI scoring may be as valid and reliable as traditional hand-scoring.

In traditional hand-scoring, a person reads an essay (or other written response), evaluates it against a rubric and other criteria (anchor papers, range-finders), and assigns a score. In AI scoring, the essay is automatically graded by a software program that uses some kind of formula (or combination of formulae) to assign the score.

Before this year, I'd always pitched my tent in a clearing in the traditional scoring camp. But that is in the best of all possible worlds, in the immortal words of Voltaire. When the rubric is fair and sound and based on observable, measurable traits; the anchors and range-finders are solid; and the hand-scorers diligent. Because don't we all have this innate aversion to the impersonal coldness of receiving a grade from a non-sentient program? A program that is not even capable of doing the thing that it is grading us on?

And yet, an experience with one of my daughters' teachers last semester made me rethink AI, at least for classroom use, at least for teachers who lack education, training, and experience in assessment. (Which so many teachers do. Assessment, though it is a big part of education, just is not adequately addressed in teacher credential programs.)

My daughters' teacher applied (or failed to apply, as she marked a project down for a trait that didn't even appear on this rubric) a rubric that broke two of the biggest rules in evaluating student performance. One was that descriptions of performance at different score point levels were exactly the same; another was that the language was completely subjective, the rubric didn't include any observations of what could be measured.

Here's a bad rubric:
Score point 4--Awesome, excellent work, student does a great job.
Score point 3--Still really great, not quite excellent.
Score point 2--Um, kind of bad, actually.
Score point 1--Weren't you listening to anything I said all semester?
Score point 0--Now you're just trying to fail. Mission accomplished, pal.

When I asked the teacher about the rubric, she defended it by saying she'd been using it for 15 years. I tried to explain that a history of use without data was no guarantee that the rubric was sound, but I'm not sure she ever understood how unfair--how invalid and unreliable--her grading system has been for 15 years. No one else ever complained. No one else knew enough to complain.

In such a case, bring on the AI. For the good of the student.

More here if you want to go on.

Tuesday, April 10, 2012

Turn Up the Heat

A friend and her 4-year-old came to visit last week. While I was making dinner, the fire alarm shrieked. The warning was on account of the scallops with bacon (I was feeling the call of the south).

Our Little Friend: What's that sound?
My First Daughter: It's the fire alarm.
Our Little Friend: What does it mean?
My First Daughter: It means Mom is cooking.
Our Little Friend: We never hear that when my mom is cooking.
My Second Daughter: Maybe she's not cooking hard enough.

As states adopt and begin implementing the Common Core standards, there's bound to be a bit of a shock. In several arenas, but just now I'm thinking of reading passages.

Rigor is what's got all the nerves in a twist at the moment, and I'm all for rigor, but it's not all we should be considering. Rigor out of context--like most anything out of context--is meaningless. One could take it to mean increased readability measure, or higher grade level vocabulary, both of which can be used to unintentionally silly effect that diminishes any credibility a text might have had without such manipulations.

In "Publishers' Criteria for the Common Core State Standards in English Language Arts and Literacy, Grades 3-12," David Coleman and Susan Pimentel describe how we--the providers of text in curriculum and assessment--need to be cooking harder in order to produce high-quality texts of sufficient complexity to meet the demands of the standards.

So there needs to be what I think of as content density and richness of ideas. Passages need to show evidence of thought and care, both in terms of the approach to the subject and the craft itself.

We're not just filling pages with print. We're giving students opportunities to learn and think and reflect and make connections and then come up with their own ideas.

UPDATE: Formatting fixes.

Tuesday, April 3, 2012


As obvious as the link is between the quality of content (we could also say validity) and item writing training, it remains a mystery to me how sound practices in item writing training have nearly become obsolete.

Steven M. Downing discusses this link in "Twelve Steps for Effective Test Development" from the Handbook of Test Development:
Yet knowing the principles of effective item writing is no guarantee of an item writer's ability to actually produce effective test questions. Knowing is not necessarily doing. Thus, one of the more important validity issues associated with test development concerns the selection and training of item writers . . . . The most essential characteristic of an effective item writer is content expertise. Writing ability is also a trait closely associated with the best and most creative item writers.
. . .
Effective item writers are trained, not born. Training of item writers is an important validity issue for test development. Without specific training, most novice item writers tend to create poor-quality, flawed, low-cognitive-level test questions that test unimportant or trivial content. Although item writers must be expert in their own disciplines, there is not reason to believe that their subject matter expertise generalizes to effective item writing expertise. Effective item writing is a unique skill and must be learned and practiced. For new item writers, it is often helpful and important to provide specific instruction using an item writer's guide, paired with a hands-on training workshop (Haladyna, 2004). As with all skill learning, feedback from expert item writers and peers is required. The instruction-practice-feedback-reinforcement loop is important for the effective development and maintenance of solid item writing skills. . . .
The best and most effective training, then, is to teach item-writing skills to content area experts, give them a guide for reference, let them practice, review and comment on their work, have them make revisions, and review and comment again until the items are satisfactory. Add a peer review step -- but only if the peers have a thorough understanding of the principles for developing sound items. Repeat as necessary.

In other news, April is National Poetry Month. I like the idea of carrying a poem in your pocket.

Monday, April 2, 2012

Send in the Clowns

This story, about the list of topics banned from New York state tests, was the circus show last week:

In a bizarre case of political correctness run wild, educrats have banned references to “dinosaurs,” “birthdays,” “Halloween” and dozens of other topics on city-issued tests.
I couldn't help commenting that it's like complaining about a fleabite when you're getting attacked by lions. One doesn't have to look far to find much more serious problems in public education.

And if it is a problem, it's one a of risk-management (as more business-minded folks would say).

Dinosaurs are banned because conservative religious groups protest any content with a whiff of a hint of a suggestion that evolution may have occurred on Earth. Halloween is banned because conservative religious groups protest references to a holiday having to do with the supernatural (ghosts, demons). Birthdays are banned because the celebrating of birthdays is prohibited by some religious groups, the same groups that would protest if tests contained references to birthdays. Danged if ye do, danged if ye don't.

In the early 1990s, the California Learning Assessment System crash and burn cost the state millions of dollars. The cause was public controversy about the assessment content.

I find all of this silly, just as I think it's nonsensical that McDonald's coffee lids have warning labels that the beverage is hot and may cause burns. But can you blame them?

All this blustering about the effect when we should be investigating the cause. But this isn't really intended to be news; it's just entertainment, as you can see from the readers who enjoy getting their rile on in the comments.