Rule Number One

...is that you gotta have fun.

Every craft, art, skill, vocation--every discipline--comes with rules. Call them procedures, steps, processes--there's a set of knowledge that must be acquired and then applied over and over.

The path to mastery is
1. learn the rules
2. practice following the rules
3. make mistakes
4. get corrected
5. do it over
6. repeat eleventy gazillion times

Until you know enough to know when you can or should break a rule, when the value of such breakage exceeds the value of following the letter of the law. When you're really the master of a craft, art, skill, vocation, you're so at home in that world that you operate on a different plane--it's all intuitive. With experience and mastery come a set of tools that aren't available to beginners or even intermediates.

The most basic foundation for the rules is that you have to care about what you're doing, which to me means you gotta have fun. There must be something in there that's absorbing or it's not worth doing. Do something else. Do something you like, something you care enough about that you'll do a good job at it. You have to like what you do so that you care about it so that you learn to do a good job.

Speaking of twenty-year-olds, Louis C.K. says to just do the job you're getting paid for. [Might not want to play this at work. Taboo words. Don't play it if you're offended by profanity.]

But I don't mean to pick on writers. I believe that many would do better if they knew how to do better. What if the writers who provide okay work and the writers who provide almost okay work want to provide excellent work? Let's assume they have the solid foundation of caring about the quality of the writing and they want to make their editor's life easier. Now what? 

There's no longer a now what in this industry. We've discussed this before. And before. There used to be a training path, but the shrubbery has overgrown the path.

Which is a long way of saying that I thought it would be good to write a series of posts to review some of the basic rules of item writing.

The test publishing industry is shrouded in mystery. First, because it's technical, and for lots of people, technical=boring. Whenever I begin to describe or explain what I do, I see all the light fade from the listener's eyes when his brain wanders off in search of sparkly entertainment. Then, because the tests are high-stakes, they are secure and confidential. Unfortunately, the security and confidentiality create an environment in which substandard work is passed off as acceptable. Who's to know? This is an industry that has incredibly high stakes for students and teachers, and yet it is completely unregulated. Think about that for a second.

The end users (those poor kids) don't know the difference. They just do what they're told. The teachers are hardly more sophisticated in this realm, and must trust their administrators, who must trust the district personnel, who must trust the state department of education, who must trust--the test publishing companies.

That whole chain might wind differently if assessment development occupied a place in teacher education--doesn't it seem strange that teachers are taught so little about what becomes of such vital importance to them and their students?

In any case, the plan is to review each of the basic rules, show what good items look like and what bad items look like, and explain what makes items good and bad. If even one teacher learned how to tell the difference between a good test item and a bad test item, that would be all to the good, and what I want is to increase the amount of good in the world.




7/23/13 10:01 p.m.  UPDATE:
After publishing this post, I received a message from a writer. I realized I may have inadvertently hurt writers' feelings. That is so not my intention!

For the sake of brevity, I used a gross description. I intended that part of the post as a bridge to another matter. My intention is to address problems in the industry. My intention is not to pick on, discourage, nor cause pain to any writer, but to consider why this industry isn't hospitable to new writers and how we might be able to change that. The industry desperately needs talent, but erects insurmountable barriers for outsiders. This industry doesn't provide training. This industry relies on very few people to produce a high volume of work for pay that gets lower and lower.

It would take another blog post to explain the matter fully. The problems are becoming legion.

Instead, I deleted part of the post. To this writer and all of the writers I had the good fortune to work with during that project:

Thank you. I learned so much from that project. It was a pleasure to work with so many talented writers who produced so many passages that were charming, fun to read, interesting, and all kinds of fantastic. I appreciate your hard work and willingness to learn. I find it unfortunate that I didn't (and don't) have the luxury to train writers at a reasonable pace, nor sufficient time to coach writers on all of the inflexible demands of the industry. That does not imply a deficiency on the part of the writer. Every writer I worked with was capable of great writing. This industry might not be the best fit for some--and that might not be a bad thing. Or possibly one day I will have the luxury to be able to coach and mentor writers so that they can succeed in this market. 

Getting Our House in Order

What is fun about homeschooling is a lot. All those lectures in the car to a captive audience? (Parents, you know what I'm talking about.) I get to make my kids take notes while I'm talking. Heheheheheheheh.


The regime schedule includes vocabulary study, a subject about which I hold strong opinions. For proficient readers, I believe vocabulary study best takes place in context, so one required activity is that the girls keep a notebook in which they write down every unfamiliar word.


Yesterday we were reading together (I was reading "The Law of Genre" by Derrida, don't ask, because anyone who knows me knows how I feel about deconstruction, but more on that in another post; Naomi was finishing Rebecca), and I came across an unfamiliar word ("liminal") and looked up the definition, and in so doing, we talked about what we thought it meant, both from the context and prior knowledge ("subliminal"). (In the same essay, I stumbled upon "invaginate." Diction is so telling, is it not. Talk about subliminal. Although my French is so poor that I'm unable to tell whether the choice of "invaginate" was necessary or creative license on the part of the translator.) And yes, "liminal" meant exactly what we thought.


In addition to keeping a record of unfamiliar words, the girls must find a definition and then write a sentence with each word, a sentence that provides enough context that would allow the reader to determine the meaning of the word.


Which meant that the topic of one of our commuting lectures was context clues, and specifically how context clues provide information about word meaning:

• restatement/definition--In which a synonym or paraphrase is provided:

All compromise is based on give and take, but there can be no give and take on fundamentals. Any compromise on mere fundamentals is a surrender. For it is all give and no take. --Mahatma Gandi [This example also uses contrast, which is described below.]

Restatement and definition cues are sometimes cued by punctuation, such as commas or parentheses:

A system of morality which is based on relative emotional values is a mere illusion, a thoroughly vulgar conception which has nothing sound in it and nothing true. -- Socrates 

• example--In which one or more illustrative examples are provided. The use of example may also be cued by punctuation: 

Ignorance more frequently begets confidence than does knowledge: it is those who know little, and not those who know much, who so positively assert that this or that problem will never be solved by science. -- Charles Darwin

• contrast--In which the opposite is provided:

There exists a kind of laughter which is worthy to be ranked with the higher lyric emotions and is infinitely different from the twitchings of a mean merrymaker.--Nikolai Gogol 

He is a wise man who does not grieve for the things he has not, but rejoices for those which he has. -- Epictetus 

As far as I'm concerned, I prefer silent vice to ostentatious virtue. -- Albert Einstein

• restriction--In which the use of other words in the sentence limits the possible meaning of a word (some like to call this the use of key words, to which I say "tomato, tomahto"):

A high station in life is earned  by the gallantry with which appalling experiences are survived with grace. -- Tennessee Williams 

It is not the strongest of the species that survives, nor the most intelligent that survives. It is the species that is the most adaptable to change. -- Charles Darwin

Of these basic categories, infinite variety may exist. That is, restriction clues may indicate relationship, such as cause and effect:

As in nature, as in art, so in grace: it is rough treatment that gives souls, as well as stones, their luster. -- Thomas Guthrie

And examples may be given as part of a system of classification:

Raptors, such as hawks, eagles, kites, and falcons, are known to be diurnal.

This is a foundational concept to ELA content development, both in terms of passage and item development. Writers of passages must build in context clues for difficult words, particularly words that are above grade level or words that have specific technical meanings; item writers must be sure to select target vocabulary words for which there are sufficient clues in the text for the reader to determine meanings.


Once this is explained, it probably seems fairly basic. You may already know all of this. But given the lack of training and lack of experience (because believe me, all you have to do in order to have the importance of this concept seared into your soul is to attend one item review committee meeting attended by master teachers with a solid grasp of the underlying principles of vocabulary acquisition) we see nowadays in content developers (already deplored previously), many item writers have no clue idea about context clues. They have no understanding of what real context clues look like, how context clues are created, nor of how readers rely on context clues -- these inexperienced or untrained item writers have no understanding that reading is a systematic process that relies on a variety of extremely complex skills, of which this is one, and even this one is very complex and employs different processes. Which means that the work they do is going to be fundamentally unsound.


When you go to build a house, you first make sure to lay a solid foundation. Make sure that house is built upon a rock.


More on vocabulary another time. I'm also big on word structure and derivatives, as you might have guessed.


P.S. Just for fun: a word frequency analyzer. You can see how language usage changes over time. Another way to while away the minutes.


UPDATE: Fixed a typo.
UDATE THE SECOND: And two others. Goodness gracious.

Speaking of Language, Cross-Referenced to Value-Added

Some item writers love multiple-choice language items; some hate them. I pitch my tent in the former category. Several years ago, I wrote all of the language items (writing conventions and writing strategies) that appeared on multiple parallel forms of a statewide high school exit examination. 

Language items may be either standalone or passage-dependent. The former are discrete entities, e.g.:

Read the sentence.

The Supreme Court may rule in favor of restrictions to freedom of speech when words are considered insendiary. 

Which is the correct spelling of the underlined word?

A incendiary *   B incendairy    C ensendiary     D Leave as is.

Passage-dependent language items accompany an editing passage as previously discussed. 

Language items may address any kind of writing skill or content knowledge targeted by an assessment: conventions (punctuation, capitalization, spelling), usage (grammar, diction), style (sentence structure and variety, diction), and organization (focus, elaboration, and support). (Rarely, writing applications skills are also assessed by multiple-choice items; one can easily see the difficulty of assessing any applied skill in this mode.)

There is some overlap (you see "diction" may fall into the usage camp or the style camp, for example, and sometimes grammar items fall into conventions, but basically, you might think of language items as mechanics and style/organization. Proofreading items then generally address mechanics, while editing and revision items may target either mechanics or style/organization, depending on the assessable skills.

Not all item writers can write language items. It's almost more of an editorly than a writerly undertaking, requiring a combination of specialized knowledge, persnicketiness, and an excellent grammar handbook. Some language items produced by unqualified writers are incomprehensible.

Sometimes these slip through the quality control cracks because language items are difficult to review because one must read very closely in order to identify that the error is an error, that the correct response is indeed correct, that the error is the type of error indicated by the standard/objective, and that there are no errors other than those intended. When we read, our brains automatically correct much of the error that we see, even when trained to do otherwise, so to review an item with intentional error that may also contain unintentional error is asking a lot. It's overwhelming, especially if you add in the editing passage, which means you have to do a lot of checking back and forth. Not to mention that a thorough knowledge of English usage and grammar is a rare commodity these days.

I have strong opinions about the content and construction of language items, to wit:

1. A language item should include only one type of error. A spelling item should not be contaminated by punctuation errors.

2. Each wrong answer choice should contain only one error.

3. The errors in the item must be the kinds of errors that students at the targeted grade level would reasonably make.

4. The errors should be obvious to the student who possesses the skill or content knowledge being assessed.

5. Trivial, why-bother sentences (or passages) should not be used to assess language skills. Language items should use actual facts in the stimulus sentences and paragraphs, rather than the easy but lame sentence.

Here is an example of an item with a trivial stimulus sentence:

Read the sentence.

 I haven't seen ______ since December.

Which pronoun should be used in the sentence?
A him *   B she    C they    D we

Here is an example of a language item based in fact:

Read the sentence.

 Langston Hughes had been writing poetry for years before Vachel Lindsay helped ____ publish his work.

 Which pronoun should be used in the sentence?

A he     B him *    C them    D they

I've written language items based on marine biology, space exploration, phenotypic plasticity, you name it. What is interesting in the world is a lot. There's no need to write about the purely meaningless.


UPDATE: Fixed some formatting with the MC items. Those are kind of tricky with the line breaks and indents.


UPDATE: Oh! I forgot to mention a recent dethspicable deplorable practice, that of using sentences from previously published (usually classic literature) as the stimulus for language items, either containing (newly imposed) embedded error or offering students options for improvements to the original (classic) writing.


Your mind is probably as boggled as mine was when first I came across this nastiness in the woodshed unspeakable horror unsound practice.


How might this work, you ask? Not well, as illustrated in the following examples.


Here is an example of the addition of error to a line from classic literature:

Read the sentence from Leo Tolstoy's Anna Karenina. 

Everyone thinks of changing the world, but no one thinks of changing themselves.

Which word would best replace the underlined word in the sentence? 

A himself*   B myself   C ourselves   D Leave as is.  

Here is an example of an invitation to students to improve upon a line from classic literature:

Read the sentence from Leo Tolstoy's Anna Karenina. 

Everyone thinks of changing the world, but no one thinks of changing himself.

To avoid repetition, which word would best replace the underlined word in the sentence? 

A altering*   B becoming   C improving   D renovating   

Why would anyone perform such sacrilege and blasphemy do such a thing? From the best of intentions.


An item writer, bored with the trivial, why-bother stimulus sentences, feels inspired to use sentences culled from the books she loves, sentences that are themselves little masterpieces of beauty, wit, and style. Won't this be good for the students? And there is no one to stop her, as these sentences are stolen borrowed from works now in the public domain.


Good intentions, road to you-know-where, cross-referenced to the law of unintended consequences.

I've Got My Reasons

. . . in the immortal but out of context words of Talib Kweli.


Speaking of intention, what if one's intention be muddled? One wouldn't get very far. But even if one does not aspire to go far, one needs to know where one is going (and though it is always lovely to leave space for beautiful surprises, it's also good to have an idea of what one might do when one gets there).


That is, intention--like levels of interpretation--can rest comfortably in the shallows or can swim in the deepest depths of the deep blue sea.


A few weeks ago, I had a conversation with a colleague about the interpretation of figurative language. As we talked, I realized I have an internal framework for the interpretation of figurative language, a framework that I used when I taught, that I use when I help my daughters with their ELA homework*, and on which I absolutely rely in my work--but that I'd never fully articulated.


At the time, I was putting finishing touches on the manuscript of a book that will be published in June, God willing and the creek don't rise, a book with a title that leaves little room for ambiguity and makes up in clarity what it lacks in evocativeness (The Hidden Market for Children's Literature: Getting Paid to Write for Reading Tests) I decided to include this part of the conversation in the book toward the interest of value-added:

If we're talking about imagery and figurative language, we would say that the progressive levels of interpretation would proceed in a manner something like this:

1. literal meaning: this is the basic, word-by-word view and is restricted to the literal definitions of the words that are then combined into sentences, lines, and paragraphs to convey literal meaning. Nothing wrong with the shallows in some circumstances. Sometimes that is the best place to be.
2. sensory-dependent meaning: creating visual images, developing rhythm through meter and structure, musicality through rhyme, assonance, repetition, etc., and the sensations of touch and taste through the use of evocative words and phrases. I see this as the body level of interpretation.
3. connotative: suggestive of shades of meanings and feelings, thereby establishing and developing tone and mood. This is emotional.
4. in terms of author's craft: contributing to and supporting the development of narrative elements(e.g., characters, setting, plot). This is artistic and intellectual. 
5. symbolic: offering motifs to represent or carry the overarching theme and other big ideas of the work. Touching the soul here.
6. extending ideas and making connections: taking the aerial view and creating a bigger, more universal picture of the human experience that may cross genders, generations, social class, cultures, historical epochs, philosophical schools/movements, etc.; making allusions to people, objects, historical events, or even cultural (including artistic or literary) periods, values, and movements. Connecting with the universal human spirit.

To return to the aerial view, when we writers are working on an assignment, sometimes our intention is so limited that it becomes wrongheaded. We forget the big idea. We think that our intention is to deliver a certain number of words assembled according to certain rules by a certain date in order to get paid. (We're not the only ones who do this; it happens in companies, too. Who hasn't seen the tension between marketing and development? Not that tension is necessarily bad--I mean it in the sense of pulling from both sides--it keeps the tent aloft.)


Some of this is necessary, right? We have to sit down, engage the mental machinery, and produce. We can't always be floating among the stars. But, and--it's well to keep in mind the ultimate goal, the real underlying purpose, which in our case is generally to give a child a fair opportunity to show us what he knows or can do in a given arena of knowledge and skill.


Because knowing this helps us to make decisions in our work that will serve that child.




UPDATE: I always think of more to say after I walk away. Esprit d'escalier. This is directly related to the deplorable tendency to repeat standard language verbatim in test items. You see? The item writer is thinking in the limited terms of providing a question that meets the specifications at the lowest possible level: You want a transition item? Here is a transition item. Again in the immortal words of that great sage Oprah, when we know better, we do better. We can do better.


UPDATE: Oh, gracious, I forgot to add the footnote to the * following "ELA homework." Here it be:
* I simply cannot believe how much homework my daughters have. I will go out on a limb and say it is immoral. Why, you may well ask. Because one strategy to compensate for inadequate instruction is to load up on the homework and in so doing, assure oneself that the kids will learn as they plod through hours of busy work at home. Sometimes my daughters have 4-6 hours of homework. They're not getting a Ph.d. They're in the eighth grade.

In Defense of Quality

If you cannot learn to love real art, at least learn to hate sham art.

This, from William Morris.

By “real art,” let’s say James Lesesne Wells, for example, and by “sham art,” let’s say Thomas Kinkade. 

I’d like to apply this sentiment to the work of writing, and more specifically, to the work of test content development. Quite frankly, I’m mystified (a phrase I borrow from an assistant district attorney with whom I used to work when I was on a very different career path, and who was in the habit of using this phrase to sharpen his tongue as he prepared to slice me up for having done something with which he disagreed) by not only the deep and devastatingly obvious diminution of quality in test content in the past few years, but also by the failure of people in this silo of the industry to recognize this trend.

Quite frankly, it breaks my heart.  As silly as it sounds. But when you love, you expose yourself to the risk of heartbreak. Again, I turn to William Morris, who said, “Give me love and work – these two only.” I’ve been doing this work for 19 years now; though I got into it thinking it was a temporary rope to keep me out of the quicksand until I found my magic circle niche place on solid ground, I think we can all agree it’s become a long-term relationship.

If this lack of quality trend were limited to newcomers to the business, we could propose that them entry-level young’uns [*sigh*] are poorly educated and ill equipped to express themselves except via texting, which you can certainly see in their editorial comments (which are lamentably rich in acronyms, emoticons, and which betray an unfortunate juvenile fondness for excess punctuation and using all caps in directions, which cannot help but set one’s back up, however patient one might be, and anyone who knows me knows that overly patient I be not).

But no, all we content dev folk – ELA, math, science, and social studies, not one is immune, no, not one – have noticed, and we do talk about it, and the conversation and all the various repetitions and iterations of the same conversation bore and horrify us so that we are reduced to shaking our heads and turning our attention to some vision of an oasis, such as the cocktail that awaits the end of the day.

Back in the day, when I worked at Great Big Huge Test Publishing Company, I went to a mandatory training on root cause analysis. We used the fishbone chart. As trainings go, it was all right. Certainly better than the one at which I was accused of not doing my work and letting my teammates pick up my slack because I failed to participate in the assembling of a puzzle, which failure actually had a lot to do with my abysmally poor spatial intelligence and equally poor vision (since corrected through the wonders of Lasik surgery) and little to do with my work ethic, which, as it happens, is about as Puritan as a work ethic might be. You can take the girl out of the working class, but you can’t take the working class out of the girl. But I digress.

If we performed a root cause analysis on the wreckage of Good Ship Quality, what would we find?

To answer that we’d have to go back to the beginning. When I started as a content editor, I was dedicated to one project. That project was my one, my only, my all in all. It was the same for my co-workers. That was the early 90s. In most states, large-scale tests were restricted to reading, writing, and math, and were administered at three or four grades (usually something like 4, 6, 8, and 10, or 5, 7, and 11).

Five years later, it was a whole different and bigger but not necessarily better ballgame. More states were testing more grades, and NCLB loomed on the horizon. As a supervisor, I was responsible for five projects. No one on my team was solely dedicated to any one project; each person, from editor to supervisor, worked on several.

I had a meeting with my manager that went like this:

Manager: [peering at her clipboard] All right, so you have State V, State W, State X, and State Y.

Me: And State Z.

Manager: Oh, I forgot about Z. Right. State Z. [scribbles a note on her clipboard]

Me: What is the order of priority?

Manager: [pause] They’re all priorities.

Me: With five states, mistakes are going to be made. It’s impossible to supervise five projects of this scale. Which state is going to be the mistake state?

NCLB  hit like an atom bomb. 

Test publishing companies couldn’t handle the workload. Companies that had never done any testing smelled the money and jumped into the fray. All companies got hiring fever. By then, I was a content development manager hiring entry-level candidates at more than twice my starting salary as an editor (and did that ever sting, I tell you what).

But the equation for meeting a production deadline is

TIME + WORKERS = MEETING PRODUCTION DEADLINE

If you have less time, you need more workers. Fewer workers, you need more time. I am no math expert, but this equation I know.

Deadlines got more and more compressed, development cycles shrank, and everyone starting skipping steps. Real training gave way to on-the-job training, which really means sink-or-swim training. New-hires were handed the comprehensive binder containing lists of processes and procedures, which binders were relegated to shelves in cubicles because no one had time to read them. Early field tests were cast aside. Sometimes all field tests. There were fewer internal reviews. The few remaining reviews were performed by overworked and/or underexperienced staff—and you can actually determine which is which (and which is both) when you see the editorial feedback coming out of such reviews.[1]

Another significant factor may be a corollary to the Peter Principle. The most highly skilled, knowledgeable, and experienced line staff keep getting promoted to management, where they may be doing a fantastic job, but their spots are filled either by new hires or old hands who are left behind (how can I say this delicately? Their remaining behind may not always be by their own choice). Combine this with the absence of training, and it’s a chaos cocktail.

Not to mention the dependence on freelancers. Companies started laying people off and then rehiring them as subcontractors. For some, it’s a win-win—the company don’t have to pay your benefits, and you get to work at home in your pajamas—but it do mean there are a heck of a lot of people at their keyboards writing test questions who neither have experience in education nor in publishing, let alone assessment, which some of us choose to believe is both an art and a science.

There is value in enduring years of slogging through the entire publishing cycle from first draft through bluelines over and over and over again. There is value in having logged many hours in the company of small children struggling to read. There is value in meeting with what the industry calls the stakeholders—teachers, administrators, community leaders, DOE officials. There is value in stretching to accommodate the demands of the stakeholders. There is value in educating oneself about the history and practices of one’s profession. Those learn-to-play-the-piano-in-10-minutes books aside, there is no shortcut to attaining mastery in anything.

There are so many facets to what we do in assessment content development, and when one’s experience is restricted to one tiny mirrored triangle of the great big disco ball, well, that creates a problem because one hasn’t constructed a greater context which allows for greater meaning to inform and guide the work. When the work is simply writing questions for a paycheck and meaning goes out the window, the questions get lamer and lamer, by which I mean trivial, superficial, and plagued by error.

However, the purpose of identifying a problem is not to castigate wrong-doers, nor to enjoy that most basic human pleasure of being right, but to use such identification to find a solution.

The answers are probably as clear to you as they are to me:

• 1.     Only hire content developers (freelance or in-house, I have no axe to grind here) who either have a proven track record of providing high-quality work or who have the capability (combination of education, writing skills, content area expertise, intelligence, creativity, and persnicketiness) to learn how to do the work well
• 2.     Provide not only adequate but excellent training
• 3.     Employ senior content development personnel [*ahem* not naming any names] to review items and provide specific instructional feedback to writers
• 4.     Budget sufficient time and money for the given project

Well, there you go. There’s nothing new under the sun, in the immortal words of King Solomon. 



2/23/12 UPDATE: This just landed in my inbox, posted by Erik Robelen at Curriculum Matters at EdWeek:

The “new normal” in educational publishing is “a severe lack of oversight in the quality of curriculum being produced” and a “frightening apathy” to do anything about it.  Keeghan’s piece, “Afraid of Your Child’s Math Textbook? You Should Be” is a jeremiad.  It does for textbook publishing what The Jungle did for the meatpacking industry. 

Keeghan paints a bleak and dispiriting picture of a business gutted by mergers, competition for fewer available dollars, and an increased focus on sales and marketing at the expense of producing quality products.  Materials rushed to market at breakneck speed are “inherently, tragically flawed.”  Plus the pool of qualified writers and editors is drying up, and those doing the work “often don’t have the necessary skills or experience to produce a text worthy of the publisher’s marketing claims,” she writes.

 You will want to read Annie Keeghan's original blog post here at Open Salon.

---

[1] Overworked but highly experienced people skim text, which forces their brain to fill in the gaps. Which means erroneous assumptions and conclusions drawn from limited evidence and resulting in unnecessary, ill-advised edits. Subtleties or fine distinctions are impossible to detect when skimming. Underexperienced people often restrict their scope of what’s acceptable to their own narrow band of direct experience, and then reject what lay in the outer darkness of their ignorance. This is bad enough on its own, but they will then assume a pedantic tone and lecture the writer for having written items that exceed the demands of the specifications.  

Fun with Point Biserials

A few weeks ago, I was spending a lot of time analyzing test data, specifically, the p values and pt. biserials of a set of English language arts tests for grades 3 through 11, in order to determine What Went Wrong.

In the preponderance of cases, of course, nothing went wrong. The items performed more or less as expected, with the correlations one might expect--i.e., high-achieving students got the easy questions and most of the hard questions right, while struggling students may have gotten the easy questions right, but pretty much got the hard questions wrong. And it follows that no wrong answer in these solid items attracted more students than the right answer.

But there were items with wacky data, items for which the high-achieving students picked wrong answers, or items with wrong answers that lured a higher number of students, leaving the right answer feeling like an awkward wallflower in a darkened gym at the middle school dance. My task was to review these items and figure out the big why. It is a testament to my thorough and absolute geekiness that I LOVE DOING THIS WORK. I could do it all day long, every day. Oh, my goodness. More fun than a barrel of monkeys. It was such a pleasure that I found it difficult to tear myself away at the end of the day (and honesty compels me to add that I did sneak back to it at night after my daughters were asleep).

Why, you may ask. Oh, for so many reasons! One being that it is fun to play detective, to deconstruct an item by conducting an investigative inquiry which concludes in determining the most likely source of the trouble, whether it be a stem of staggering verbosity, or a fundamental unsoundness in the premise of the item, or a bad practice (e.g., attempting to assess two or more skills with one item). It's like picking up a big tangled knotted mess of yarn and, starting with one end, delicately unraveling the knots and twists and then rolling the yarn back into a nice orderly ball. But with your brain. How often do we really have the opportunity to use our brains in this way, and truly, what can possibly be more satisfying that solving a problem?

Because assessment content development, as its own universe, is governed by its own rules, and--to me, in my ridiculous and relentless geekiness--these rules have a sort of simple elegance, the beauty of rigor and orderliness. (Were I to wax biblical on the matter, I would start talking about the necessity for doing things decently and in order.) So to use the data as a mirror to reflect the soundness (or unsoundness) of the item, diagnose the trouble, and then find a means of repair requires the use of a range of tools that include a knowledge of the rules (and knowing when it is acceptable to bend them) and of how students might interpret an item (which is not always how a test developer intends) and of the content area itself. And it also requires a bit of intuition, that ability to sense your way in the dark that just comes from thoroughly knowing something, in the same manner you could navigate your home with the lights off because you know where the couch is and where stands that big coffee table with the treacherously sharp corners.

But Is It Art?

A couple of years ago, I spent an entire month writing stories, articles, and poems for a shelf product for a company that shall remain nameless. What could be more fun than that? With the poems, I amused myself by choosing different forms in which to write. The children who later read the poems had no idea they were reading a villanelle or a poem in terza rima, but the discipline of a strict form gave me license to be more creative with both content and language.

There's a kind of beauty in rigor and discipline; you can see this in any creative endeavor, including item writing. I've seen items that were elegant in their simplicity and adherence to the demands of the form. Following all the rules doesn't make for a perfect item--even a flatly literal basic recall item may be written perfectly, and yet be a bad item. But it's impossible to be ignorant of (or ignore) the rules and write a good item.

That having been said, when you have completely mastered the rules, you may want to break one or two once in a while, for a specific purpose. The catch is, of course, that the flouting of the rule must be intentional and purposeful, or it just looks like what it is: a mistake.