What Are We Waiting for

November is National Novel Writing Month (NaNoWriMo, that crazy venture in which you write 50,000 words of a consecutive narrative thread in 30 days). The NaNoWriMo motto is The world needs your novel.

I don't know whether the world needs my novel, but I do know that every November for the last 5 years, I considered participating in NaNoWriMo but didn't follow through.

This year I am doing NaNoWriMo. Or I should say I did NaNoWriMo, because I'm done.

In the first 7 days I wrote 20,661 words. As of midnight yesterday, I had written 50,425 words, which means I finished the novel and accomplished what I had set out to do. (Almost. I'd set myself the ridiculous goal of 50,000 words in 15 days. So I was a day late.) 

NaNoWriMo critics will agree with Laura Miller of Salon who says "I am not the first person to point out that 'writing a lot of crap' doesn’t sound like a particularly fruitful way to spend an entire month, even if it is November." (To which one might be tempted to respond thusly.) I would agree with Laura Miller that if a writer is using some of the silly tricks I've seen for completing NaNoWriMo, such as adding a nonsense word ("potato") after every word in order to pad the word count, it's potato pretty potato much potato a why-bother potato exercise potato.

Certainly writing fast doesn't always mean writing badly. Mental Floss provides a list of NaNoWriMo novels that got published.

As for me, I found the NaNoWriMo project a delightful and fruitful way to spend two weeks and a day.

If you are a writer who has considered participating in NaNoWriMo, I encourage you to do it.

Because if your novel offers the world your unique voice, your quirky perspective, your way of looking at and thinking about the world, and your story--the story only you can tell--then yes, the world really does need your novel.

We need all the stories we can get.

PSA: Conducting a Performance Review of the Inner Critic

For a few days I've been considering what Freud identified as the super-ego, what has become popularly known as the inner critic, or if you're abundantly blessed, your personal busload or committee or parliament of inner critics.

Generally speaking, the role of the inner critic seems primarily protective, as wrongheaded as its tactics are for that purpose. If you're a writer, your inner critic might advise you to clean the house instead of writing, or may suggest there's no point in writing, because everything you write is horrible, bad, and no-good, and that one 7th grade English teacher was just being nice or had bad taste or that the one story you wrote that got published or that your writing group liked was an anomaly. No attempt means no risk means safety from rejection and criticism. Probably there are inner critics for every occupation.

There are as many strategies of dealing with inner critics as there are types of inner critics: we can embrace them, treat them with compassion, or murder them. I'd like to offer a new one: conduct a performance review to evaluate their effectiveness on the job.

Frankly, if my inner critics were my employees, I'd have fired them by now. I always sort of knew they were making a mess of it--how did they even make it past the interview?-- but I'd never taken the time to consider the nature of their incompetence. I made one of the biggest mistakes employers make with performance reviews, according to Forbes: I haven't been conducting them.

To prepare for the review, I must specify the responsibilities of the inner critic: what is the job description?

A critic approaches a work with curiosity, is open to experiencing a work and responding to the work, observes those responses, and investigates the work in order to consider the artist's purpose and message, and how the artist achieves that purpose and conveys that message. A critic pays close attention to the work. A critic is dedicated to a particular art (and should be an expert of that art or field), and in the raising up of that art, and so the critic is responsible for celebrating achievements as much as for observing attempts that may fall short.

These are questions a critic might ask in examining a work of art:

• What place does this work have in the tradition or genre?
• How has this work been influenced by earlier works?
• How does this work reflect a social, historical, or cultural context?
• How may different levels of interpretation be applied to this work?
• What are the messages of this work?
• Is there a universal message or is the message specific to a group, time, place, instance?
• How does this work convey those messages?
• Does this work have a unique voice and style that set it apart from others of its type?
• What contributes to the uniqueness of this work?
• What emotional responses does a person have to this work?
• How does this work elicit those emotional responses?

What is not in the critic's job description:

• insulting, bullying, ridiculing, mocking, belittling, name-calling, disparaging the artist or the work, listing past mistakes, predicting future failure

Critics address the work with specific questions and use the information to discuss the work and explore the meaning of the work. It's a lot easier to approach a work from opinion (I like it, I hate it) or judgment (rotten, all right, awesome), but so much less interesting and less effective.

What good does opinion do, really? Even nice opinions aren't much use, as much as it feels better when a reader says that he likes one's writing than when he says that he hates it.

If you heard the way my inner critics talk, you'd agree that they've fallen down on the job. Or maybe they got handed the wrong job description in the first place. I can't recall their ever having performed satisfactorily.

Writers need to write, and we need to figure out ways around and through and over and under the obstacles to writing, even while we're constructing the obstacles.

And there are other reasons:

• We parents, teachers, coaches, aunts, uncles, grandparents--we all of us adults--create the voices for future generations of inner critics. I'd like my kids to go out into the world with inner critics who contribute to, rather than sabotage, their happiness and well-being.
• Many of us supervise others. In so doing, we can't help but impose the messages of our inner critics on others. We have the choice to be constructive or destructive. We have the choice to be reasonable (or not) with ourselves and others.
• No good writing ever came from performance anxiety.

I'm concerned that how we think about, talk about, teach, and measure performance in writing trains students to believe that not only are they terrible writers, but they are not capable of writing well. The reality is that everyone can tell a good story, and if you can tell a good story, you can write a good story, and if you can write a good story, you should, because we all need stories, as many as possible, from as many perspectives as possible.

The Big Idea, or Focus, Cross-Referenced to Basic Rules of Item Writing

What comes before preparation is intention, which we previously discussed here. Still, the concept of the Big Idea bears further exploration.

Let's consider how we might approach this grade 4 standard from the CCSS, RL.4.2:

Determine a theme of a story, drama, or poem from details in the text; summarize the text.

This standard is passage-dependent; students read a story, poem, or play (or excerpts of the same) and then answer questions about what they read.

This standard requires two distinct subskills: determining a theme and summarizing text. 

Either may be assessed with multiple-choice, constructed-response, or technology-enhanced items, although I note that in an ideal world, we wouldn't use multiple-choice for summarizing, but would instead ask students to create the summary. Again in that ideal world, it's best if we provide the student with opportunities to demonstrate mastery of a particular skill by allowing the student to perform the skill; however, we often operate under constraints that exclude the ideal. That's okay.

After we've read all of our ancillary support materials and have thoroughly acquainted ourselves with the story, poem, or play (for less experienced item writers and for all item writers without a strong background in literary analysis, I suggest making an outline of and annotating the passage in order to avoid the trap of writing superficial and repetitive items), we determine the theme(s). There may be more than one. Out of fairness, choose the strongest theme that is most clearly supported and most thoroughly developed in the passage. The theme may be stated explicitly or may be implied by the characters' words and actions.

Here is our passage, "A Boy's Song" by James Hogg.

    Where the pools are bright and deep,
    Where the gray trout lies asleep,
    Up the river and o'er the lea,
    That's the way for Billy and me.

    Where the blackbird sings the latest,
    Where the hawthorn blooms the sweetest,
    Where the nestlings chirp and flee,
    That's the way for Billy and me.

    Where the mowers mow the cleanest,
    Where the hay lies thick and greenest,
    There to trace the homeward bee,
    That's the way for Billy and me.

    Where the hazel bank is steepest,
    Where the shadow falls the deepest,
    Where the clustering nuts fall free.
    That's the way for Billy and me.

    Why the boys should drive away,
    Little sweet maidens from the play,
    Or love to banter and fight so well,
    That's the thing I never could tell.

    But this I know, I love to play,
    Through the meadow, among the hay;
    Up the water and o'er the lea,
    That's the way for Billy and me.

We would probably use call-out boxes to define some of the vocabulary--"lea" and "nestling" stand out as words likely to interfere with student understanding.

If we're writing a multiple-choice item, the stem will look like this:

What is a theme of the poem?

Or we might identify the poem only by its title ("What is a main theme of 'A Boy's Song'?") if we plan to write another item about genre characteristics ("How does the reader know 'A Boy's Song' is a poem?").

Often at the lower grades, we use "theme" and "main idea" as synonyms; depending on curriculum, grade 4 students may not yet be familiar with the specific terms for narrative elements, and we don't want to erect unnecessary obstacles for those students, so we might write a stem that looks like this:

What is a main idea of the story?

I prefer "a" rather than "the" in order to allow for variety in literary interpretation; we'd follow the client's preference on this. In this case, a clear theme is the joy of spending time in nature. Now we have a stem and the correct response:

What is a theme of the poem?
A the joy of spending time in nature
B [TK]
C [TK]
D [TK]

Next we'd write three distractors (wrong answers). Each distractor should have a rationale--that is, each should embody a specific mistake or breakdown in comprehension or literary analysis that might hinder a student en route to determining the theme. The rule in item writing is that, given the evidence in the text, distractors must be "plausible but not possible." The distractors should be clearly wrong to the student who is able to "determine a theme...from details in the text."

Many clients require item writers to provide rationales or justifications for the wrong answer; I support this wholeheartedly as valuable practice for inexperienced item writers. Experienced item writers have rationales in their minds already, so it's just a matter of typing them.

When we write the distractors, we must stay focused on our Big Idea. In order to do that, we'd consider the breakdowns that occur when students attempt to identify a theme. In order to do that, we'd think about the process of making meaning from text. We read the poem and step back and come up with the overarching meaning: the joy of spending time in nature. Then we think about how a student might falter in putting the pieces of the poem together to see that big picture. A student might get stuck on a detail of the poem, and mistake that for a theme. A student might confuse theme and subject. A student might focus too narrowly.

Next up: constructing plausible but not possible distractors.

What I'm reading: The Reivers by Faulkner and Imaginings of Sand by Andre Brinks.

The Big Idea, Cross-Referenced to Basic Rules of Item Writing

What endeavor doesn't benefit from planning and preparation? What endeavor succeeds without preparation?

Ah, fatal words! Too late in moving here, too late in arriving there, too late in coming to this decision, too late in starting with enterprises, too late in preparing.

These first guidelines of the CCSSO/TILSA Quality Control Checklist for Item Development and Test Form Construction should be considered in early stages of planning, long before item writing assignments are made:

1A. Each item should assess content standard(s) as specified in the test blueprint or assessment frameworks. 

2A. Items must measure appropriate thinking skills as specified in the test blueprint or assessment frameworks. 

3A. Items should be written at appropriate cognitive levels and reading levels according to the item specifications guidelines.

A test blueprint identifies the skills and/or knowledge to be assessed, provides the item-to-skill distribution, and specifies item formats.

Let's think about creating a blueprint to assess writing at grade 6. We'll base the blueprint on the Common Core State Standards.

In the CCSS, English conventions are addressed in the language standards, and what we might call writing strategies and application are addressed in the writing standards. The language standards could be assessed with a variety of formats: standalone or passage-dependent multiple choice items, standalone or passage-dependent technology-enhanced items, or as one component of an extended-constructed-response item.

Here is a writing standard:

1. Write arguments to support claims in an analysis of substantive topics or texts,using valid reasoning and relevant and sufficient evidence.a. Introduce precise claim(s), distinguish the claim(s) from alternate oropposing claims, and create an organization that establishes clearrelationships among claim(s), counterclaims, reasons, and evidence. 

b. Develop claim(s) and counterclaims fairly, supplying evidence for eachwhile pointing out the strengths and limitations of both in a manner thatanticipates the audience’s knowledge level and concerns. 

c. Use words, phrases, and clauses to link the major sections of the text,create cohesion, and clarify the relationships between claim(s) and reasons,between reasons and evidence, and between claim(s) and counterclaims. 

d. Establish and maintain a formal style and objective tone while attending tothe norms and conventions of the discipline in which they are writing. 

e. Provide a concluding statement or section that follows from and supportsthe argument presented.

Generally the above standard would be assessed with an extended-constructed-response item, because multiple-choice items and short constructed-response items don't allow students sufficient opportunity to demonstrate the ability to "write arguments to support claims...." However, the subskills may be (and frequently are) assessed with multiple-choice items; this is more common at the district or classroom level than at the state level. You might see a question that addresses W.1.a by asking the student to choose the best opposing claim for a given argument. Such multiple-choice items may help teachers isolate specific areas in which a student needs instruction and support.

Here are language standards:

1. Demonstrate command of the conventions ofstandard English grammar and usage whenwriting or speaking. 

a. Ensure that pronouns are in the proper case(subjective, objective, possessive). 

b. Use intensive pronouns (e.g., myself,ourselves). 

c. Recognize and correct inappropriate shifts inpronoun number and person.* 

d. Recognize and correct vague pronouns(i.e., ones with unclear or ambiguousantecedents).* 

e. Recognize variations from standard Englishin their own and others’ writing andspeaking, and identify and use strategies toimprove expression in conventional language.* 

2. Demonstrate command of the conventions ofstandard English capitalization, punctuation, andspelling when writing. 

a. Use punctuation (commas, parentheses,dashes) to set off nonrestrictive/parentheticalelements.* 

b. Spell correctly.

All of the above language skills may be assessed with multiple-choice questions. These could be standalone, or could offer a stimulus: an editing passage with embedded errors. More on language items as previously discussed here.

For our imaginary grade 6 writing test, we might decide that we'd like to use multiple measures in order to obtain as much information as possible in as many different ways as we can, so we're going to create a blueprint that specifies a combination of item formats and includes x number of multiple-choice and technology-enhanced items, along with one extended-constructed-response to a writing prompt; this response will be scored with a holistic rubric that addresses organization, style and voice, and conventions. We would develop a test blueprint that specified the standards and subskills to be assessed, along with the number of items and item formats for each standard or subskill.

In our blueprint, we may also use Bloom's Taxonomy or Norman Webb's Depth of Knowledge Guide to determine the cognitive level for each item. Although the cognitive levels of some skills are relatively simple to determine, based on what is required from students, some skills may be addressed at multiple levels of cognitive complexity.

We may instead indicate the cognitive levels, item difficulty, and content or domain limits, and reading levels in the item specifications, as suggested in the CSSO/TILSA checklist.

In a typical statewide high-stakes assessment program, the decisions that inform the development of a test blueprint and item specifications are made by committees, which is as it should be, and committees should include classroom teachers. Committees often include other stakeholders, e.g., business leaders who may be asked to identify skills and knowledge necessary in the workplace.

Once all of that preparation is complete, item development begins.

Now let's say we've received an assignment to write those multiple-choice language items and that ECR writing prompt. We've read all of the project documentation and support materials; we have the item specifications in front of us. 

It is a truth universally acknowledged that a test item should target one and only one skill or bit of content knowledge. Each idea should have one big idea; every part of the item should support that focus. 

If we were going to write a multiple-choice item for W.2.b, our big idea would be how to spell grade-level appropriate words. We might write an item that looks like this:

Which word is spelled correctly?
A absense
B boundery
C civilizashion
D dissolve*

This item clearly targets one skill: correctly spell grade-level-appropriate words. The stem tells the student exactly what to do. The item is phrased simply and concisely. The content is neutral; there are no highly-charged words. All of the answer choices are grade 6 words (according to EDL Core Vocabularies); all are words likely to be known to grade 6 students and are words that are significant to academic content areas. There are no tricky or esoteric rare words. The answer choices appear in a logical order (here we use alpha order). All of the distractors address common spelling mistakes: using s instead of c, using e instead of a, and writing phonetically. None of the words are homonyms and so none are context-dependent; each of these words have one correct spelling.

Here is a poor item addressing the same skill:

Which word is written correctly?
A musheenz
B rabby
C anker
D pistol

This item has multiple flaws. First, the big idea is not specified in the stem; the student doesn't know what s/he is expected to do until s/he reads the answer choices. The answer choices are not grade-level-appropriate; "machine" is a grade 2 word, while "anchor" is grade 3. The word "rabbi" may not be familiar to grade 6 students. Answer choice A ("musheenz") is plural, while the other ACs are singular. Answer choice A also offers mistakes that are unlikely to be made by students at the targeted grade level. The answer choices do not appear in any logical order. Finally, the correct response is a type of weapon.

As bad as this item is, though, we could make it even worse by

• increasing the reading load by burying the spelling words in sentences and offering four sentences as the answer choices;
• obscuring the targeted skill by adding in other types of conventions errors, such as mistakes in capitalization and punctuation;
• using homonyms, or words that are spelled differently depending on the context;
• using above-grade-level vocabulary.

Item writing is both an art and a science. There's so much to consider, even in writing the simplest spelling item.

How to Get the Best from Item Writers

Many years ago, I was a development manager at a Great Big Huge Test Publishing Company. I've already told the story of how I began as a temp employee in hand-scoring, as so many recruits to the test publishing industry do. Armed with my book-learnin' and a new but hardly marketable M.A. in English, emphasis in creative writing, I was thrilled to get a job that paid slightly more than $10 an hour, a job that had to do with words and writing. Yay me, illustrating the joy of low expectations.

When the Great Big Huge Test Publishing Company was awarded what was then considered a big statewide assessment contract (back in the days when we tested at grades 3, 6, 8, and 10, or grades thereabouts), I was plucked from hand-scoring, handed the title of associate editor and deposited in a cubicle in a cavernous upstairs honeycomb which cubicle I shared with another associate editor who'd also come from hand-scoring. Within 5 years, I'd gone from the windowless cubicle of associate editor to content editor to supervisor to program manager to the window office of development manager. You can probably guess at my success as a manager, given I had no training and little experience in management. Oh, if only I had read Bringing Out the Best in People: How to Apply the Astonishing Power of Positive Reinforcement.

Which I have since read, and which principles I endeavor to apply when I'm called to supervise others, and to the effectiveness of which I can attest. We do what we know; when we know better, we do better.

It may be tempting, when we consider the plummeting quality of what we see on test materials, to blame the writers. But as this video from ETS reminds us, the writer is only one of many contributors.

Getting the best from item writers has to start long before an editor sends out that Are you available? email. The foundation of a project must be sound; there must be a blueprint and prototypes; there must be a clear vision of what the product is intended to look like, how it is intended to perform, what skills/knowledge it is intended to measure, and how it should measure those skills/knowledge.

These decisions should not be left to the item writer; few item writers are equipped to make such decisions. In the past, item writers worked in-house, or were mostly former employees of test publishing companies, and so were at least minimally conversant with principles governing the design and construction of assessments. Sometimes item writers were corralled to help assemble the tests and were given Xeroxed sheets containing lists of item numbers and associated data. That is no longer the case. I don't know of any test publishing company who maintains a staff of in-house item writers. Today test publishing companies commonly hire item writers who have never worked for test publishing companies and who have little experience writing items for high-stakes assessment (they may have written for curriculum and textbooks, if they have any experience at all). They may not have any classroom experience; they may not even have kids, and so the world of education--the real world of education and of what kids really are able to know and do at a given grade level--is a mystery to them. Or they develop their own ideas about what K-12 students know and can do, ideas that are as inaccurate as they are ambitious and inflated. (This is through no fault of their own, but the remedy is simple: volunteer in the classroom. Go to a school and offer to spend an hour a week in a classroom.)

Even if item writers were equipped, they shouldn't make decisions which should rightfully be made at a much higher level, by folks with greater knowledge, experience, and authority. Such decisions take time. There must be time to consider, reflect, think about it in the shower and in the car, time to return to one's colleagues and say Well, what if and how will it work if. The what-ifs must be given time to rise to the surface.

Rushing inevitably creates chaos. Whatever writers produce under slippery circumstances--when the expectations are not specified-- will fail to meet those unspecified expectations.

Assuming, however, that the big decisions have been made, and that the writers have been provided with everything they need to do a good (or excellent) job, what else can companies do to get the best from writers?

1. Take care of all housekeeping details upfront. Provide the writer with written information about the scope of work, schedule, deadlines, pay rates, and points of contact. Preferably all in one email message. Send the contract and the W-9. Tell the writer whom to invoice and how. Remove possible sources of worry. Worry is destructive to creativity and productivity. 
2. Provide training. The training should be as brief as possible, and should be conducted at the commencement of the project. A training that is offered a month before writing begins is useless, because writers will have forgotten the information they learned. Materials for the training should be emailed in advance. The writers should be told whom to call if they have questions.
3. There should be a dedicated content lead available to respond to writers' questions and to provide timely guidance throughout the course of the project.
4. Give writers the chance to do it right. The content lead's ducks must be lined up and ready to waddle. There must be a clear style to follow, preferences to comply with, and so on. The directions and feedback should be clear. To be effective, feedback must be immediate. Feedback must have the purpose of informing work in progress. Consider how disheartening it is to submit 50 items and then be told that there is now a new requirement, please revise those items accordingly and resubmit.
5. Allow the writers to work as they work best. More and more companies are requiring writers to input items directly into an online authoring system. While some of these are better than others, all add time and effort on the part of the writer, thus siphoning off energy better spent on item development. For each project, writers must learn how to use a new system; they might finish the project before they become proficient. Then it's off to a new system. I often decline opportunities to work in authoring systems, because I find the levels of clickage annoying--seconds add up to minutes add up to hours over the course of a year, hours I would much rather have spent reading or looking out the window or talking to my daughters or whatever else.
6. Let the writers do the work they do best. Writers write. Now that companies are operating on principles of leanness akin to corporate anorexia, companies are expecting writers to take on the work that used to be the province of content editors and desk-top publishers. With no increase in pay and no increase in time allotted to do the work.
7. Give writers enough space to write. Some assignments are so rigid and exacting, with so many criteria of so many types, that they become impossible.
8. Allow the writers to contribute their unique knowledge, experience, and skills. Writers work for all the educational assessment, test preparation, and curriculum publishers. They have access to a depth and breadth of knowledge about what's happening in educational publishing that is denied to the folks whose only job in educational publishing has been to work at the one company at which they are currently employed. Being open to the possibility that the writers know something and giving the writers freedom beyond the stricture This is how we do it will only serve the company and ultimately, the kids.
9. Be a human and let the writer be a human. We are none of us robots. We all have strengths and weaknesses. We have the skills we shine at and the skills we don't. This is normal and the nature of being human; it's not a flaw unique to a particular writer if she has trouble juggling multiple spreadsheets (not to name any names, me). No one in this world is capable of doing everything perfectly; no one is guilty of never making a mistake. The industry used to understand that; the protocol for test publishing included many rounds of editorial review prior to submitting materials to proofreading, and then to QA. 

If these principles were applied, quality would improve.

That's all I got for today. I'm off to Valencia, to the Cal Arts campus, to visit my daughter, Twin A ("A" being the initial written on the knitted cap the nurses placed on her head after her birth) who is a creative writer in the California State Summer School for the Arts program. 

What I'm reading: I finished As I Lay Dying. I love Faulkner. It always takes me at least half the book to marshall my resources to focus on his writing, I find it so challenging, but once I'm in, I'm there. I have a novel by Andre Brinks next, I think.

A Great Deal Done Imperfectly

Better a little which is well done, than a great deal imperfectly.--Plato

In my last post, I may have given the impression that editors have more power than they do, and that perhaps that they have time to consider the ramifications of failing to provide all the necessities, or that they are willfully negligent. Salaried as they may be, editors often find themselves in an unenviable pickle, with compressed development cycles and few resources. The industry's reliance on freelance personnel increases the workload of front-line staff, who may now have to manage groups of writers in addition to performing other duties. Each writer must add about an hour a week in emails, phone calls, and admin tasks--and that's if the writer is low-maintenance.

It's also likely that editors want to provide all the necessities, but those necessities don't exist and the schedule doesn't allow time for editors to develop them. (Some of the most experienced item writers are able to work around the deficiencies, but the work of the less experienced will be affected.)

No one--except the one at the top of the pyramid, I imagine-- is resting on a velvet cushion.

I may have left another inaccurate impression: that it's all about the money. It's not. How can it be? This is not a high rolling game. What I mean to say is that when writers don't have what they need to do their best work, everyone loses.

The industry continues to become less hospitable to the people actually doing the work of creating the tests--or, more accurately, the people writing the passages and questions from which the tests are assembled--which results in a great deal done imperfectly.

Writers lose time and money; they also lose the best of all rewards, the satisfaction of a job well done, simply because how can you do a task perfectly when the task hasn't been clearly defined, and when you ask for clarification, you're directed to figure it out?

The companies lose much, much more. The lower pay and the more pain (inconvenience? Call it what you will. I mean all of those tiny ducks that are pecking us to death) to the writers, the lower quality the work, and the fewer writers willing to undertake that work, those fewer writers being the ones who have no choice: the least proficient, the least experienced. And the most highly skilled writers simply decide they've had enough and they move on to greener (or at least different) pastures.

Most importantly, the children who are taking the tests have already lost when they're faced with low-quality materials that don't provide them with a fair chance to demonstrate what they know and can do.

All right. Let's move on. I'm eager to address the basic rules of item writing (a version of which you can see here, in the Quality Control Checklist published by CCSSO), but I realize I should first define some terms.

An item is a test question. An item may be discrete, or may depend on some external stimulus, such as a reading passage or a chart or a map or something else.

Here is a discrete item:

Why does my dog Sophie bark at mail carriers? 

A She is flat-out crazy.

B She is outraged by uninvited guests.*

C She knows something about them that we don't. 

D She wants to register a protest about mail delays.

The above is a multiple-choice question, and contains a stem ("Why does my dog bark at mail carriers?") and four answer choices: one correct response (B, as far as I can tell, but I think maybe C is a possible right answer) and three distractors. Distractors, which used to be known as "foils," are wrong answers. Don't get hung up on the language--the point is never to distract nor entice the test-taker to bubble the wrong answer; the point is to create wrong answers that have a reasonable foundation in common mistakes kids would make with that particular skill or bit of content knowledge. More on this later. But tests should never be tricky.

A multiple-choice item is usually worth one score point, and used to be budgeted for one minute of test-taking time, not including the time it takes to read a passage or examine whatever stimuli is needed to answer the question.

There are other item formats: constructed-response items, which are also known as open-ended items. These require the student to provide a response. The response may be as short as a word or a phrase, or, in the case of extended-constructed-response items, the response may be a complete essay.

Here is a short constructed-response item:

Write two words to describe my dog Sophie. Use details to support your answer.

And here is the scoring rubric:

2 points: The response includes two accurate describing words, and is supported by relevant evidence. 

1 point: The response includes one accurate describing word, and is supported by relevant evidence, OR the response includes two accurate describing words with no supporting evidence. 

0 points: The response is blank, illegible, off-topic, or otherwise impossible to score.

A short constructed-response item would usually have a score point range of 0-2 or 0-3, and would be budgeted for 5-10 minutes. More than that is usually reserved for an ECR, which could take as few as 15 minutes, or as long as an hour or more for a full essay.

An extended-constructed-response item would look like this:

Considering Sophie's protective nature, do you think it is wise for strangers to approach her? Why or why not? Write an essay in which you discuss the wisdom of approaching a dog with whom you are personally unacquainted.

I don't provide a writing rubric because they are complex creations, but you may see some examples here and here. The score point ranges for ECR items vary, depending on the traits of writing and number of domains. That is, an essay might be scored for organization, style, and conventions. If the question depends on the student's comprehension of a passage, the essay might be scored for both reading and writing.

Bear in mind that these sample items are jokes, and as such, aren't examples of exemplary items, primarily because they require a great deal of prior knowledge, and so the test-taker who is unfamiliar with Sophie and dogs in general will perform less well than the test-taker who is on a first-name basis with Sophie and/or other dogs. There are other, less egregious flaws, but we'll get to those when we get to them.

If you have an item you'd like me to examine, explain, or deconstruct, feel free to post it in the comments. Check the copyright first.

 What I'm reading: Forgot to mention I was also finishing up The Claverings by Anthony Trollope. Then it's back to As I Lay Dying. I gave up on the other.

Before the Beginning

In the beginning, when all is without form and void, here is what an item writer should have in order to write test items:

• project overview and background
• company and/or project style guide, whichever should be followed for the assignment
• list of standards to be assessed (with or without item-to-skill distribution)
• item specifications with content and domain limits
• sample items
• scoring criteria
• boilerplate text for rubrics or sample rubrics for CR and ECR items
• item templates, if required, or log-in information for online authoring systems
• if an online authoring system is required, writers should be provided with training and a handbook or at the least, FAQ
• additional information as to client preferences (e.g., should target vocabulary be at or slightly above grade level, are there particular constructs or formats that should be included or avoided, should developers use EDL Core Vocabulary or Children's Writer's Word Book to verify grade-level-appropriateness of vocabulary)

If these documents are not available, the project is not ready to launch content development. These are not luxuries; these are the bare minimum. These should be provided with every assignment. Without these, the best case scenario is that the writer operates at a significant disadvantage and loses work time puzzling and attempt to read the minds of the assigning editor.

Remember, item writers are usually paid by the item. We aren't paid for puzzling and mind-reading. Nor are we paid to pore over these materials--which poring takes four to eight hours if you do it right; editors often don't consider that each project is its own world with its own lexicon and laws, nor do they understand the need for ramp-up time, because to them, this is all old hat, they've already spent hours and hours studying the project and discussing the project in meetings, heck, they may have contributed to the proposal and participated in a series of internal and external start-up meetings--but we all of us content developers accept that it's an unfortunate cost of doing business and we wish the test publishing company personnel understood that each project comes with unpaid ramp-up time--four hours may not be a lot to those who are comfortably salaried with medical benefits, vacation days, sick leave, and retirement accounts, but to we who are not and we who have no vacation days and no sick days, we who are paid only as long as we are clicking away at our keyboard--and that is why we decline those small assignments for fewer than 50 items. (I long for the day when every content developer performed a cost/benefits analysis, and began declining work that costs him or her money. If we all got together, the world would change. It would have to.)

The need for puzzling and mind-reading may not be quite so troubling to the assigning editors--many of whom have never written items themselves (doesn't that seem funny? It's true! Oh, maybe they wrote one sample that one time for a proposal) and so don't know what item writers need, and also, some of whom have regrettably been taught by management ("Have the item writers do it! That's what they're paid for!") to view content developers as recalcitrant underlings, sort of grumbling lower housemaids, if you will--but will be troubling them in a worse scenario, one in which the writer is unable to meet specifications because none have been provided. So the writer submits items that are unusable for the project, and the editor either rejects or rewrites the items. Either way, the schedule is compromised, and now the editor feels the pain, too.

The item writer also must be provided with a clearly defined assignment that includes the following information:

• targeted grade level(s)
• item formats (multiple-choice, constructed-response, writing prompts, technology-enhanced, etc.)
• additional criteria for formats, such as whether the items will be presented online or as paper-and-pencil tests
• number of items
• guidance about item-to-skill distribution if the distribution is not specified
• mode of delivery (Word templates delivered through email, documents uploaded to a secure FTP, files added to Dropbox, submission through an authoring system, etc.)
• point of contact--whom to contact with questions and best means of contact
• deadline
• any expectations not already specified in the project documentation
• housekeeping details: whom to invoice, what information should be included, when to invoice

What is pleasant, but not necessary, is the editor's set of item review criteria or the item evaluation checklist; however, few companies have such a rubric in place, and so the items may be reviewed with a level of subjectivity that doesn't serve either party.

This is all before the beginning.

If any of this sounds unfamiliar--if you have questions about any of this, please ask me. Ask me. I will explain every little bit. Knowledge = power. 

What I'm reading: As I Lay Dying by William Faulkner, current New Yorker, and The Shortest Way Home by Juliette Fay (not such a big fan of the latter--has anyone read it? Can you convince me to continue? I'm thinking of doing the unthinkable and stopping in the middle. The reviews seem hyperbolic for what's there.)

UPDATE: Look what I found. A sneak preview, click here. More TK!
UPDATE: Made a correction.

Rule Number One

...is that you gotta have fun.

Every craft, art, skill, vocation--every discipline--comes with rules. Call them procedures, steps, processes--there's a set of knowledge that must be acquired and then applied over and over.

The path to mastery is
1. learn the rules
2. practice following the rules
3. make mistakes
4. get corrected
5. do it over
6. repeat eleventy gazillion times

Until you know enough to know when you can or should break a rule, when the value of such breakage exceeds the value of following the letter of the law. When you're really the master of a craft, art, skill, vocation, you're so at home in that world that you operate on a different plane--it's all intuitive. With experience and mastery come a set of tools that aren't available to beginners or even intermediates.

The most basic foundation for the rules is that you have to care about what you're doing, which to me means you gotta have fun. There must be something in there that's absorbing or it's not worth doing. Do something else. Do something you like, something you care enough about that you'll do a good job at it. You have to like what you do so that you care about it so that you learn to do a good job.

Speaking of twenty-year-olds, Louis C.K. says to just do the job you're getting paid for. [Might not want to play this at work. Taboo words. Don't play it if you're offended by profanity.]

But I don't mean to pick on writers. I believe that many would do better if they knew how to do better. What if the writers who provide okay work and the writers who provide almost okay work want to provide excellent work? Let's assume they have the solid foundation of caring about the quality of the writing and they want to make their editor's life easier. Now what? 

There's no longer a now what in this industry. We've discussed this before. And before. There used to be a training path, but the shrubbery has overgrown the path.

Which is a long way of saying that I thought it would be good to write a series of posts to review some of the basic rules of item writing.

The test publishing industry is shrouded in mystery. First, because it's technical, and for lots of people, technical=boring. Whenever I begin to describe or explain what I do, I see all the light fade from the listener's eyes when his brain wanders off in search of sparkly entertainment. Then, because the tests are high-stakes, they are secure and confidential. Unfortunately, the security and confidentiality create an environment in which substandard work is passed off as acceptable. Who's to know? This is an industry that has incredibly high stakes for students and teachers, and yet it is completely unregulated. Think about that for a second.

The end users (those poor kids) don't know the difference. They just do what they're told. The teachers are hardly more sophisticated in this realm, and must trust their administrators, who must trust the district personnel, who must trust the state department of education, who must trust--the test publishing companies.

That whole chain might wind differently if assessment development occupied a place in teacher education--doesn't it seem strange that teachers are taught so little about what becomes of such vital importance to them and their students?

In any case, the plan is to review each of the basic rules, show what good items look like and what bad items look like, and explain what makes items good and bad. If even one teacher learned how to tell the difference between a good test item and a bad test item, that would be all to the good, and what I want is to increase the amount of good in the world.




7/23/13 10:01 p.m.  UPDATE:
After publishing this post, I received a message from a writer. I realized I may have inadvertently hurt writers' feelings. That is so not my intention!

For the sake of brevity, I used a gross description. I intended that part of the post as a bridge to another matter. My intention is to address problems in the industry. My intention is not to pick on, discourage, nor cause pain to any writer, but to consider why this industry isn't hospitable to new writers and how we might be able to change that. The industry desperately needs talent, but erects insurmountable barriers for outsiders. This industry doesn't provide training. This industry relies on very few people to produce a high volume of work for pay that gets lower and lower.

It would take another blog post to explain the matter fully. The problems are becoming legion.

Instead, I deleted part of the post. To this writer and all of the writers I had the good fortune to work with during that project:

Thank you. I learned so much from that project. It was a pleasure to work with so many talented writers who produced so many passages that were charming, fun to read, interesting, and all kinds of fantastic. I appreciate your hard work and willingness to learn. I find it unfortunate that I didn't (and don't) have the luxury to train writers at a reasonable pace, nor sufficient time to coach writers on all of the inflexible demands of the industry. That does not imply a deficiency on the part of the writer. Every writer I worked with was capable of great writing. This industry might not be the best fit for some--and that might not be a bad thing. Or possibly one day I will have the luxury to be able to coach and mentor writers so that they can succeed in this market.