The Big Idea, Cross-Referenced to Basic Rules of Item Writing

What endeavor doesn't benefit from planning and preparation? What endeavor succeeds without preparation?

Ah, fatal words! Too late in moving here, too late in arriving there, too late in coming to this decision, too late in starting with enterprises, too late in preparing.

These first guidelines of the CCSSO/TILSA Quality Control Checklist for Item Development and Test Form Construction should be considered in early stages of planning, long before item writing assignments are made:

1A. Each item should assess content standard(s) as specified in the test blueprint or assessment frameworks. 

2A. Items must measure appropriate thinking skills as specified in the test blueprint or assessment frameworks. 

3A. Items should be written at appropriate cognitive levels and reading levels according to the item specifications guidelines.

A test blueprint identifies the skills and/or knowledge to be assessed, provides the item-to-skill distribution, and specifies item formats.

Let's think about creating a blueprint to assess writing at grade 6. We'll base the blueprint on the Common Core State Standards.

In the CCSS, English conventions are addressed in the language standards, and what we might call writing strategies and application are addressed in the writing standards. The language standards could be assessed with a variety of formats: standalone or passage-dependent multiple choice items, standalone or passage-dependent technology-enhanced items, or as one component of an extended-constructed-response item.

Here is a writing standard:

1. Write arguments to support claims in an analysis of substantive topics or texts,using valid reasoning and relevant and sufficient evidence.a. Introduce precise claim(s), distinguish the claim(s) from alternate oropposing claims, and create an organization that establishes clearrelationships among claim(s), counterclaims, reasons, and evidence. 

b. Develop claim(s) and counterclaims fairly, supplying evidence for eachwhile pointing out the strengths and limitations of both in a manner thatanticipates the audience’s knowledge level and concerns. 

c. Use words, phrases, and clauses to link the major sections of the text,create cohesion, and clarify the relationships between claim(s) and reasons,between reasons and evidence, and between claim(s) and counterclaims. 

d. Establish and maintain a formal style and objective tone while attending tothe norms and conventions of the discipline in which they are writing. 

e. Provide a concluding statement or section that follows from and supportsthe argument presented.

Generally the above standard would be assessed with an extended-constructed-response item, because multiple-choice items and short constructed-response items don't allow students sufficient opportunity to demonstrate the ability to "write arguments to support claims...." However, the subskills may be (and frequently are) assessed with multiple-choice items; this is more common at the district or classroom level than at the state level. You might see a question that addresses W.1.a by asking the student to choose the best opposing claim for a given argument. Such multiple-choice items may help teachers isolate specific areas in which a student needs instruction and support.

Here are language standards:

1. Demonstrate command of the conventions ofstandard English grammar and usage whenwriting or speaking. 

a. Ensure that pronouns are in the proper case(subjective, objective, possessive). 

b. Use intensive pronouns (e.g., myself,ourselves). 

c. Recognize and correct inappropriate shifts inpronoun number and person.* 

d. Recognize and correct vague pronouns(i.e., ones with unclear or ambiguousantecedents).* 

e. Recognize variations from standard Englishin their own and others’ writing andspeaking, and identify and use strategies toimprove expression in conventional language.* 

2. Demonstrate command of the conventions ofstandard English capitalization, punctuation, andspelling when writing. 

a. Use punctuation (commas, parentheses,dashes) to set off nonrestrictive/parentheticalelements.* 

b. Spell correctly.

All of the above language skills may be assessed with multiple-choice questions. These could be standalone, or could offer a stimulus: an editing passage with embedded errors. More on language items as previously discussed here.

For our imaginary grade 6 writing test, we might decide that we'd like to use multiple measures in order to obtain as much information as possible in as many different ways as we can, so we're going to create a blueprint that specifies a combination of item formats and includes x number of multiple-choice and technology-enhanced items, along with one extended-constructed-response to a writing prompt; this response will be scored with a holistic rubric that addresses organization, style and voice, and conventions. We would develop a test blueprint that specified the standards and subskills to be assessed, along with the number of items and item formats for each standard or subskill.

In our blueprint, we may also use Bloom's Taxonomy or Norman Webb's Depth of Knowledge Guide to determine the cognitive level for each item. Although the cognitive levels of some skills are relatively simple to determine, based on what is required from students, some skills may be addressed at multiple levels of cognitive complexity.

We may instead indicate the cognitive levels, item difficulty, and content or domain limits, and reading levels in the item specifications, as suggested in the CSSO/TILSA checklist.

In a typical statewide high-stakes assessment program, the decisions that inform the development of a test blueprint and item specifications are made by committees, which is as it should be, and committees should include classroom teachers. Committees often include other stakeholders, e.g., business leaders who may be asked to identify skills and knowledge necessary in the workplace.

Once all of that preparation is complete, item development begins.

Now let's say we've received an assignment to write those multiple-choice language items and that ECR writing prompt. We've read all of the project documentation and support materials; we have the item specifications in front of us. 

It is a truth universally acknowledged that a test item should target one and only one skill or bit of content knowledge. Each idea should have one big idea; every part of the item should support that focus. 

If we were going to write a multiple-choice item for W.2.b, our big idea would be how to spell grade-level appropriate words. We might write an item that looks like this:

Which word is spelled correctly?
A absense
B boundery
C civilizashion
D dissolve*

This item clearly targets one skill: correctly spell grade-level-appropriate words. The stem tells the student exactly what to do. The item is phrased simply and concisely. The content is neutral; there are no highly-charged words. All of the answer choices are grade 6 words (according to EDL Core Vocabularies); all are words likely to be known to grade 6 students and are words that are significant to academic content areas. There are no tricky or esoteric rare words. The answer choices appear in a logical order (here we use alpha order). All of the distractors address common spelling mistakes: using s instead of c, using e instead of a, and writing phonetically. None of the words are homonyms and so none are context-dependent; each of these words have one correct spelling.

Here is a poor item addressing the same skill:

Which word is written correctly?
A musheenz
B rabby
C anker
D pistol

This item has multiple flaws. First, the big idea is not specified in the stem; the student doesn't know what s/he is expected to do until s/he reads the answer choices. The answer choices are not grade-level-appropriate; "machine" is a grade 2 word, while "anchor" is grade 3. The word "rabbi" may not be familiar to grade 6 students. Answer choice A ("musheenz") is plural, while the other ACs are singular. Answer choice A also offers mistakes that are unlikely to be made by students at the targeted grade level. The answer choices do not appear in any logical order. Finally, the correct response is a type of weapon.

As bad as this item is, though, we could make it even worse by

• increasing the reading load by burying the spelling words in sentences and offering four sentences as the answer choices;
• obscuring the targeted skill by adding in other types of conventions errors, such as mistakes in capitalization and punctuation;
• using homonyms, or words that are spelled differently depending on the context;
• using above-grade-level vocabulary.

Item writing is both an art and a science. There's so much to consider, even in writing the simplest spelling item.

Jobs, Money, and Love for All

Good grammar and spelling skills make your life better.

From Education Week, a teacher's blog post about an employer who screens job candidates to weed out those with poor English grammar skills:

In a fiery post for the Harvard Business Review, Wiens says he flat out won't hire people who are careless with grammar. And to ensure that no offenders slip through, both of his companies—Wiens is also the founder of the documentation-software maker Dozuki—have instituted mandatory grammar tests as part of the hiring process.

Before I read the source post by Kyle Wiens for Harvard Business Review, I had mixed feelings. Is this really necessary for candidates who aren't writers? I know so many smart, capable people who use--oh, let's call it "nonstandard grammar." They're not writers, and so I switch off the inner critic before I read their email messages.

But Wiens makes a compelling argument:

Grammar signifies more than just a person's ability to remember high school English. I've found that people who make fewer mistakes on a grammar test also make fewer mistakes when they are doing something completely unrelated to writing — like stocking shelves or labeling parts. 

In the same vein, programmers who pay attention to how they construct written language also tend to pay a lot more attention to how they code. You see, at its core, code is prose. Great programmers are more than just code monkeys; according to Stanford programming legend Donald Knuth they are "essayists who work with traditional aesthetic and literary forms." The point: programming should be easily understood by real human beings — not just computers. 

And just like good writing and good grammar, when it comes to programming, the devil's in the details. In fact, when it comes to my whole business, details are everything. 

He's right that details matter.

You can even put a price on details. According to a 2004 article in The New York Times, one spelling mistake can cost as much as $198:

Mr. Green once bought a box of gers for $2. They were gears for pocket watches, which he cleaned up and put back on the auction block with the right spelling. They sold for $200. 

An article published by BBC News in 2011 calculates the cumulative costs:

An online entrepreneur says that poor spelling is costing the UK millions of pounds in lost revenue for Internet businesses.

Wayne State University assistant professor Fred Vultee, who conducted a study sponsored by the American Copy Editors' Society, found that not only can readers distinguish edited from unedited text, but they notice and are troubled by errors (more here). Said Vultee, "Editing makes a difference. It's across the board, it's not imaginary, and it's reasonably big."

And if that weren't enough, errors in spelling and grammar can also inhibit romance: "In a survey of more than 5,000 Match.com members asking about the most common profile mistakes, 51% complained about profiles with poor spelling or grammar...."

Over the last almost twenty years, I've worked with writers whose English language skills range from impeccable to not. When I say "not," I don't mean to pick on anyone who makes an occasional typo or some other minor mistake. It's very difficult to proof one's own work; the brain insists on making the text appear as you know it should appear. I use "not" for those whose work displays not just a repeated pattern of error, but sometimes repeated patterns of different types of errors, even after those same types of errors have been pointed out to the writer previously.

Is it prejudice to expect that people whose trade is writing should have strong language skills? Maybe. For me, it's work avoidance; writers who make lots of mistakes cause me lots of extra work. Which means I can verify that mistakes have a cost, because if I need writers, my first calls are to the ones whose writing causes me the least work in terms of content and style.

I don't fault anyone for not knowing something; we all of us stumble on our ignorance sometimes. And yet, it works against us to cling to that ignorance once we become aware of it.

Toward the goal of jobs, money, and love for all, I offer the following resources:
Grammar & Style: Grammar Guides, Style Guides, APA, MLA--LibrarySpot.com
Grammar.net
HyperGrammar at the University of Ottawa
On Writing Well by William Zinsser
Pam Nelson: Grammar Guide
The Elements of Style, Fourth Edition by William Strunk, E.B. White, and Roger Angell
The Purdue Online Writing Lab
Top Ten Resources on Spelling and Word Study

Why Not Offer Miracles If You Can?

Miracles

WHY! who makes much of a miracle?
As to me, I know of nothing else but miracles,
Whether I walk the streets of Manhattan,
Or dart my sight over the roofs of houses toward the sky,
Or wade with naked feet along the beach, just in the edge of the water,	5
Or stand under trees in the woods,
Or talk by day with any one I love—or sleep in the bed at night with any one I love,
Or sit at table at dinner with my mother,
Or look at strangers opposite me riding in the car,
Or watch honey-bees busy around the hive, of a summer forenoon,	10
Or animals feeding in the fields,
Or birds—or the wonderfulness of insects in the air,
Or the wonderfulness of the sun-down—or of stars shining so quiet and bright,
Or the exquisite, delicate, thin curve of the new moon in spring;
Or whether I go among those I like best, and that like me best—mechanics, boatmen, farmers,	15
Or among the savans—or to the soiree—or to the opera,
Or stand a long while looking at the movements of machinery,
Or behold children at their sports,
Or the admirable sight of the perfect old man, or the perfect old woman,
Or the sick in hospitals, or the dead carried to burial,	20
Or my own eyes and figure in the glass;
These, with the rest, one and all, are to me miracles,
The whole referring—yet each distinct, and in its place.
To me, every hour of the light and dark is a miracle,
Every cubic inch of space is a miracle,	25
Every square yard of the surface of the earth is spread with the same,
Every foot of the interior swarms with the same;
Every spear of grass—the frames, limbs, organs, of men and women, and all that concerns them,
All these to me are unspeakably perfect miracles.
To me the sea is a continual miracle;	30
The fishes that swim—the rocks—the motion of the waves—the ships, with men in them,
What stranger miracles are there?                                                   Walt Whitman

"To me, every hour of the light and dark is a miracle." To me, this poem is a miracle.


I could spend hours looking out the window and thinking about this. I did spend hours thinking about the poem at night as I walked my dog in the canyon and looked up at the stars and heard the breeze brush through the palms and in the morning as I took my daughters to school and we crested the hill and caught sight of the ocean and the islands and in the evening as I took out the trash and saw the sunset so garish that if it were a painting, you'd make fun of the artist.


I came across this poem because recently I had the great good fortune of having the opportunity to think about how to bring the American Romantics into certain high school classrooms in a certain state (intentional vagueness required by both professional courtesy and the stipulations of the non-disclosure agreement that is part of my contract).


For a week, I read Emerson, Lake, and Palmer Emerson ("To believe your own thought, to believe that what is true for you in your private heart is true for all men, — that is genius"), Whitman, Dickinson, Dunbar. Leaving out Thoreau wasn't intentional; it's just that I started with Emerson, but once I was in the thick of the poetry, decided to stay there and tackle the essays another time.


This work was pure pleasure. Poetry and grade 11 are a match made in heaven. All those emotions, for one thing. And there's a poet for everyone. Baudelaire was the original Emo:

Dark one, I am torn
By your savage ways,
Then, soft as the moon, your gaze
Sees my tortured heart reborn.

                                      --from "Afternoon Song"

Not the least of the pleasure was my thinking about the students. Maybe there would be one or two who for whom these poems might be a signpost to the onramp to the highway that leads to this gorgeous world of poetry and self-knowledge.


Last week, I went to hear a friend and fellow alumnus from the College of Creative Studies at UCSB  speak about literature and read from his writing. He told the students about how, when he graduated, he had this sense of having a gift, a talent, that was bigger than he knew what to do with.


It was true of all of us, I think. I'll go further and say that it may be true of everyone, but maybe everyone is not lucky enough to understand that he has a gift, or not lucky enough to land in a place where he has the room and space and encouragement to find and exercise his gift.


My daughters sweetly and patiently listen when I talk about my work and endlessly quote from my readings. They have more than a passing familiarity with Emerson by now. Erin, my eldest, showed me what her social studies textbook says about Emerson and Thoreau.


The targeted standard is to read the writings of the American Transcendentalists. The means employed by the textbook writer to address this standard was to write one spare paragraph about these two in which Thoreau is described as someone who was jailed because he refused to pay a one-dollar federal tax and Emerson is described as someone who didn't want to go to jail and so he paid the tax. That's it. Thus summing up the philosophies and values of American Transcendentalism. There was no context, no real biographical information, there were no excerpts from the essays.

Why in the name of all that is holy would a writer (or a publisher, as surely it was not solely the writer's decision--as a writer, I often bump into the decisions of editors and publishers) pass up the opportunity to offer more?

In the immortal words of Emily Dickinson (see below), the brain is wider than the sky. Even at 14. I'm sure their brains could not only accommodate bushels, nay, truckloads of real information, rather than a dismissive tag line. Why not offer the miracles and let the students sort them out?

CXXVI THE BRAIN is wider than the sky, For, put them side by side, The one the other will include With ease, and you beside. The brain is deeper than the sea, For, hold them, blue to blue, The one the other will absorb, As sponges, buckets do. The brain is just the weight of God, For, lift them, pound for pound, And they will differ, if they do, As syllable from sound.

UPDATE: Fixed link to College of Creative Studies at UCSB.