The Big Idea, or Focus, Cross-Referenced to Basic Rules of Item Writing

What comes before preparation is intention, which we previously discussed here. Still, the concept of the Big Idea bears further exploration.

Let's consider how we might approach this grade 4 standard from the CCSS, RL.4.2:

Determine a theme of a story, drama, or poem from details in the text; summarize the text.

This standard is passage-dependent; students read a story, poem, or play (or excerpts of the same) and then answer questions about what they read.

This standard requires two distinct subskills: determining a theme and summarizing text. 

Either may be assessed with multiple-choice, constructed-response, or technology-enhanced items, although I note that in an ideal world, we wouldn't use multiple-choice for summarizing, but would instead ask students to create the summary. Again in that ideal world, it's best if we provide the student with opportunities to demonstrate mastery of a particular skill by allowing the student to perform the skill; however, we often operate under constraints that exclude the ideal. That's okay.

After we've read all of our ancillary support materials and have thoroughly acquainted ourselves with the story, poem, or play (for less experienced item writers and for all item writers without a strong background in literary analysis, I suggest making an outline of and annotating the passage in order to avoid the trap of writing superficial and repetitive items), we determine the theme(s). There may be more than one. Out of fairness, choose the strongest theme that is most clearly supported and most thoroughly developed in the passage. The theme may be stated explicitly or may be implied by the characters' words and actions.

Here is our passage, "A Boy's Song" by James Hogg.

    Where the pools are bright and deep,
    Where the gray trout lies asleep,
    Up the river and o'er the lea,
    That's the way for Billy and me.

    Where the blackbird sings the latest,
    Where the hawthorn blooms the sweetest,
    Where the nestlings chirp and flee,
    That's the way for Billy and me.

    Where the mowers mow the cleanest,
    Where the hay lies thick and greenest,
    There to trace the homeward bee,
    That's the way for Billy and me.

    Where the hazel bank is steepest,
    Where the shadow falls the deepest,
    Where the clustering nuts fall free.
    That's the way for Billy and me.

    Why the boys should drive away,
    Little sweet maidens from the play,
    Or love to banter and fight so well,
    That's the thing I never could tell.

    But this I know, I love to play,
    Through the meadow, among the hay;
    Up the water and o'er the lea,
    That's the way for Billy and me.

We would probably use call-out boxes to define some of the vocabulary--"lea" and "nestling" stand out as words likely to interfere with student understanding.

If we're writing a multiple-choice item, the stem will look like this:

What is a theme of the poem?

Or we might identify the poem only by its title ("What is a main theme of 'A Boy's Song'?") if we plan to write another item about genre characteristics ("How does the reader know 'A Boy's Song' is a poem?").

Often at the lower grades, we use "theme" and "main idea" as synonyms; depending on curriculum, grade 4 students may not yet be familiar with the specific terms for narrative elements, and we don't want to erect unnecessary obstacles for those students, so we might write a stem that looks like this:

What is a main idea of the story?

I prefer "a" rather than "the" in order to allow for variety in literary interpretation; we'd follow the client's preference on this. In this case, a clear theme is the joy of spending time in nature. Now we have a stem and the correct response:

What is a theme of the poem?
A the joy of spending time in nature
B [TK]
C [TK]
D [TK]

Next we'd write three distractors (wrong answers). Each distractor should have a rationale--that is, each should embody a specific mistake or breakdown in comprehension or literary analysis that might hinder a student en route to determining the theme. The rule in item writing is that, given the evidence in the text, distractors must be "plausible but not possible." The distractors should be clearly wrong to the student who is able to "determine a theme...from details in the text."

Many clients require item writers to provide rationales or justifications for the wrong answer; I support this wholeheartedly as valuable practice for inexperienced item writers. Experienced item writers have rationales in their minds already, so it's just a matter of typing them.

When we write the distractors, we must stay focused on our Big Idea. In order to do that, we'd consider the breakdowns that occur when students attempt to identify a theme. In order to do that, we'd think about the process of making meaning from text. We read the poem and step back and come up with the overarching meaning: the joy of spending time in nature. Then we think about how a student might falter in putting the pieces of the poem together to see that big picture. A student might get stuck on a detail of the poem, and mistake that for a theme. A student might confuse theme and subject. A student might focus too narrowly.

Next up: constructing plausible but not possible distractors.

What I'm reading: The Reivers by Faulkner and Imaginings of Sand by Andre Brinks.

Rule Number One

...is that you gotta have fun.

Every craft, art, skill, vocation--every discipline--comes with rules. Call them procedures, steps, processes--there's a set of knowledge that must be acquired and then applied over and over.

The path to mastery is
1. learn the rules
2. practice following the rules
3. make mistakes
4. get corrected
5. do it over
6. repeat eleventy gazillion times

Until you know enough to know when you can or should break a rule, when the value of such breakage exceeds the value of following the letter of the law. When you're really the master of a craft, art, skill, vocation, you're so at home in that world that you operate on a different plane--it's all intuitive. With experience and mastery come a set of tools that aren't available to beginners or even intermediates.

The most basic foundation for the rules is that you have to care about what you're doing, which to me means you gotta have fun. There must be something in there that's absorbing or it's not worth doing. Do something else. Do something you like, something you care enough about that you'll do a good job at it. You have to like what you do so that you care about it so that you learn to do a good job.

Speaking of twenty-year-olds, Louis C.K. says to just do the job you're getting paid for. [Might not want to play this at work. Taboo words. Don't play it if you're offended by profanity.]

But I don't mean to pick on writers. I believe that many would do better if they knew how to do better. What if the writers who provide okay work and the writers who provide almost okay work want to provide excellent work? Let's assume they have the solid foundation of caring about the quality of the writing and they want to make their editor's life easier. Now what? 

There's no longer a now what in this industry. We've discussed this before. And before. There used to be a training path, but the shrubbery has overgrown the path.

Which is a long way of saying that I thought it would be good to write a series of posts to review some of the basic rules of item writing.

The test publishing industry is shrouded in mystery. First, because it's technical, and for lots of people, technical=boring. Whenever I begin to describe or explain what I do, I see all the light fade from the listener's eyes when his brain wanders off in search of sparkly entertainment. Then, because the tests are high-stakes, they are secure and confidential. Unfortunately, the security and confidentiality create an environment in which substandard work is passed off as acceptable. Who's to know? This is an industry that has incredibly high stakes for students and teachers, and yet it is completely unregulated. Think about that for a second.

The end users (those poor kids) don't know the difference. They just do what they're told. The teachers are hardly more sophisticated in this realm, and must trust their administrators, who must trust the district personnel, who must trust the state department of education, who must trust--the test publishing companies.

That whole chain might wind differently if assessment development occupied a place in teacher education--doesn't it seem strange that teachers are taught so little about what becomes of such vital importance to them and their students?

In any case, the plan is to review each of the basic rules, show what good items look like and what bad items look like, and explain what makes items good and bad. If even one teacher learned how to tell the difference between a good test item and a bad test item, that would be all to the good, and what I want is to increase the amount of good in the world.




7/23/13 10:01 p.m.  UPDATE:
After publishing this post, I received a message from a writer. I realized I may have inadvertently hurt writers' feelings. That is so not my intention!

For the sake of brevity, I used a gross description. I intended that part of the post as a bridge to another matter. My intention is to address problems in the industry. My intention is not to pick on, discourage, nor cause pain to any writer, but to consider why this industry isn't hospitable to new writers and how we might be able to change that. The industry desperately needs talent, but erects insurmountable barriers for outsiders. This industry doesn't provide training. This industry relies on very few people to produce a high volume of work for pay that gets lower and lower.

It would take another blog post to explain the matter fully. The problems are becoming legion.

Instead, I deleted part of the post. To this writer and all of the writers I had the good fortune to work with during that project:

Thank you. I learned so much from that project. It was a pleasure to work with so many talented writers who produced so many passages that were charming, fun to read, interesting, and all kinds of fantastic. I appreciate your hard work and willingness to learn. I find it unfortunate that I didn't (and don't) have the luxury to train writers at a reasonable pace, nor sufficient time to coach writers on all of the inflexible demands of the industry. That does not imply a deficiency on the part of the writer. Every writer I worked with was capable of great writing. This industry might not be the best fit for some--and that might not be a bad thing. Or possibly one day I will have the luxury to be able to coach and mentor writers so that they can succeed in this market.