Showing posts with label language. Show all posts
Showing posts with label language. Show all posts

Wednesday, July 31, 2013

The Big Idea, Cross-Referenced to Basic Rules of Item Writing

What endeavor doesn't benefit from planning and preparation? What endeavor succeeds without preparation?
Ah, fatal words! Too late in moving here, too late in arriving there, too late in coming to this decision, too late in starting with enterprises, too late in preparing.
These first guidelines of the CCSSO/TILSA Quality Control Checklist for Item Development and Test Form Construction should be considered in early stages of planning, long before item writing assignments are made:
1A. Each item should assess content standard(s) as specified in the test blueprint or assessment frameworks. 
2A. Items must measure appropriate thinking skills as specified in the test blueprint or assessment frameworks. 
3A. Items should be written at appropriate cognitive levels and reading levels according to the item specifications guidelines.

A test blueprint identifies the skills and/or knowledge to be assessed, provides the item-to-skill distribution, and specifies item formats.

Let's think about creating a blueprint to assess writing at grade 6. We'll base the blueprint on the Common Core State Standards.

In the CCSS, English conventions are addressed in the language standards, and what we might call writing strategies and application are addressed in the writing standards. The language standards could be assessed with a variety of formats: standalone or passage-dependent multiple choice items, standalone or passage-dependent technology-enhanced items, or as one component of an extended-constructed-response item.

Here is a writing standard:
1. Write arguments to support claims in an analysis of substantive topics or texts,using valid reasoning and relevant and sufficient evidence.a. Introduce precise claim(s), distinguish the claim(s) from alternate oropposing claims, and create an organization that establishes clearrelationships among claim(s), counterclaims, reasons, and evidence. 
b. Develop claim(s) and counterclaims fairly, supplying evidence for eachwhile pointing out the strengths and limitations of both in a manner thatanticipates the audience’s knowledge level and concerns. 
c. Use words, phrases, and clauses to link the major sections of the text,create cohesion, and clarify the relationships between claim(s) and reasons,between reasons and evidence, and between claim(s) and counterclaims. 
d. Establish and maintain a formal style and objective tone while attending tothe norms and conventions of the discipline in which they are writing. 
e. Provide a concluding statement or section that follows from and supportsthe argument presented.


Generally the above standard would be assessed with an extended-constructed-response item, because multiple-choice items and short constructed-response items don't allow students sufficient opportunity to demonstrate the ability to "write arguments to support claims...." However, the subskills may be (and frequently are) assessed with multiple-choice items; this is more common at the district or classroom level than at the state level. You might see a question that addresses W.1.a by asking the student to choose the best opposing claim for a given argument. Such multiple-choice items may help teachers isolate specific areas in which a student needs instruction and support.

Here are language standards:
1. Demonstrate command of the conventions ofstandard English grammar and usage whenwriting or speaking. 
a. Ensure that pronouns are in the proper case(subjective, objective, possessive). 
b. Use intensive pronouns (e.g., myself,ourselves). 
c. Recognize and correct inappropriate shifts inpronoun number and person.* 
d. Recognize and correct vague pronouns(i.e., ones with unclear or ambiguousantecedents).* 
e. Recognize variations from standard Englishin their own and others’ writing andspeaking, and identify and use strategies toimprove expression in conventional language.* 
2. Demonstrate command of the conventions ofstandard English capitalization, punctuation, andspelling when writing. 
a. Use punctuation (commas, parentheses,dashes) to set off nonrestrictive/parentheticalelements.* 
b. Spell correctly.

All of the above language skills may be assessed with multiple-choice questions. These could be standalone, or could offer a stimulus: an editing passage with embedded errors. More on language items as previously discussed here.

For our imaginary grade 6 writing test, we might decide that we'd like to use multiple measures in order to obtain as much information as possible in as many different ways as we can, so we're going to create a blueprint that specifies a combination of item formats and includes x number of multiple-choice and technology-enhanced items, along with one extended-constructed-response to a writing prompt; this response will be scored with a holistic rubric that addresses organization, style and voice, and conventions. We would develop a test blueprint that specified the standards and subskills to be assessed, along with the number of items and item formats for each standard or subskill.

In our blueprint, we may also use Bloom's Taxonomy or Norman Webb's Depth of Knowledge Guide to determine the cognitive level for each item. Although the cognitive levels of some skills are relatively simple to determine, based on what is required from students, some skills may be addressed at multiple levels of cognitive complexity.

We may instead indicate the cognitive levels, item difficulty, and content or domain limits, and reading levels in the item specifications, as suggested in the CSSO/TILSA checklist.

In a typical statewide high-stakes assessment program, the decisions that inform the development of a test blueprint and item specifications are made by committees, which is as it should be, and committees should include classroom teachers. Committees often include other stakeholders, e.g., business leaders who may be asked to identify skills and knowledge necessary in the workplace.

Once all of that preparation is complete, item development begins.

Now let's say we've received an assignment to write those multiple-choice language items and that ECR writing prompt. We've read all of the project documentation and support materials; we have the item specifications in front of us. 

It is a truth universally acknowledged that a test item should target one and only one skill or bit of content knowledge. Each idea should have one big idea; every part of the item should support that focus. 

If we were going to write a multiple-choice item for W.2.b, our big idea would be how to spell grade-level appropriate words. We might write an item that looks like this:

Which word is spelled correctly?
A absense
B boundery
C civilizashion
D dissolve*

This item clearly targets one skill: correctly spell grade-level-appropriate words. The stem tells the student exactly what to do. The item is phrased simply and concisely. The content is neutral; there are no highly-charged words. All of the answer choices are grade 6 words (according to EDL Core Vocabularies); all are words likely to be known to grade 6 students and are words that are significant to academic content areas. There are no tricky or esoteric rare words. The answer choices appear in a logical order (here we use alpha order). All of the distractors address common spelling mistakes: using s instead of c, using e instead of a, and writing phonetically. None of the words are homonyms and so none are context-dependent; each of these words have one correct spelling.

Here is a poor item addressing the same skill:

Which word is written correctly?
A musheenz
B rabby
C anker
D pistol

This item has multiple flaws. First, the big idea is not specified in the stem; the student doesn't know what s/he is expected to do until s/he reads the answer choices. The answer choices are not grade-level-appropriate; "machine" is a grade 2 word, while "anchor" is grade 3. The word "rabbi" may not be familiar to grade 6 students. Answer choice A ("musheenz") is plural, while the other ACs are singular. Answer choice A also offers mistakes that are unlikely to be made by students at the targeted grade level. The answer choices do not appear in any logical order. Finally, the correct response is a type of weapon.

As bad as this item is, though, we could make it even worse by

  • increasing the reading load by burying the spelling words in sentences and offering four sentences as the answer choices;
  • obscuring the targeted skill by adding in other types of conventions errors, such as mistakes in capitalization and punctuation;
  • using homonyms, or words that are spelled differently depending on the context;
  • using above-grade-level vocabulary.
Item writing is both an art and a science. There's so much to consider, even in writing the simplest spelling item.






Saturday, August 18, 2012

What the World Needs Now

It may be unnecessary to say that in talking about what's wrong, I don't mean to pick on anybody in particular. Even if it is one person making specific mistakes, I tend to see this as a systemic ill, as previously discussed, rather than an opportunity to dogpile on an unwitting offender.

A systemic ill that is particularly pernicious in the test publishing industry. How can we expect kids to do well on tests when the tests themselves are riddled with error?

The safeguards are so simple:
1. provide adequate training (again, as previously discussed)
2. follow some kind of standard process for detecting and repairing error
3. use the information from #2 to supplement #1

These safeguards cost time and money, which explains why so many companies have gradually let them fall by the wayside. 

Training takes time and attention. I can't believe how fortunate I was to have received the training I did. In my first year as an associate editor, I probably had more training opportunities than editors now receive in a decade. It was a different era. The Great Big Huge Test Publishing Company at which I worked had an army of style editors, proofreaders, and QA personnel.

Over the last dozen years, I've seen companies completely dismantle their style editing departments and outsource the work to freelance editors. Now style editors (copy editors) are held in such low regard that if any Bright Young Thing with a talent for grammar and language conventions tells me she'd like to give this work a try, I steer her away from style editing and into content.  I hate doing it. I myself hold style editors in high esteem, maybe because often their job seems to be to keep me from making a fool of myself. What the world needs now--no disrespect intended to Dionne Warwick and Burt Bacharach--is more style editing.

Many companies have also dismantled their content development departments.

So there's no real continuity for item writers and content editors. They work on a project for one company and then skip to a project for another. The content development people who remain at the companies are overwhelmed and certainly don't have the time to provide training to freelancers; editors accept or reject items, which means they're likely to fix mistakes themselves, and so the item writer never hears of his mistakes and is free to repeat those mistakes forever in a state of unfortunate ignorant complacence.

Right now, I'm thinking of all this in terms of language conventions items, because I was reviewing a set this morning, but I'm sure this applies to all domains and content areas.

Right now, I offer some specific observations about language items:
1. When language items accompany an editing passage that contains embedded error, the content of each item must be mutually exclusive. A sentence from the passage should contain only one error, and should be used only for one item in a set.
2. "Syntax" has to do with grammatical rules; "diction" has to do with the writer's choice of words.
3. An item should clearly target only one type of error, and should not mix types of errors. For example, a punctuation item shouldn't include distractors that contain errors in verb tense.

More previously discussed here.

As an aside, it's interesting how you can learn a lot about a person by reviewing his or her work, and how this body of knowledge grows over time. It's like how my mechanic told me that he knows more about his customers from the condition of their cars than he does from talking with them. He can tell who drives erratically, who slams on the brakes, who grinds the gears, and he can tell who doesn't. Everything is everything. You can develop a similar personality profile simply by reviewing a writer's work.

I can't tell you how many times I've finished working on a manuscript, items or reading passage, don't make no difference, and felt a deep and tender affection for the writer because his or her work demonstrated care, attention, and conscientiousness. 

UPDATE: Forgot to mention. I started writing about all this because I saw this book mentioned in the Common Core Standards and immediately ordered it. 

Wednesday, March 14, 2012

Speaking of Language, Cross-Referenced to Value-Added

Some item writers love multiple-choice language items; some hate them. I pitch my tent in the former category. Several years ago, I wrote all of the language items (writing conventions and writing strategies) that appeared on multiple parallel forms of a statewide high school exit examination. 

Language items may be either standalone or passage-dependent. The former are discrete entities, e.g.:
Read the sentence.
The Supreme Court may rule in favor of restrictions to freedom of speech when words are considered insendiary. 
Which is the correct spelling of the underlined word?
A incendiary *   B incendairy    C ensendiary     D Leave as is.

Passage-dependent language items accompany an editing passage as previously discussed. 

Language items may address any kind of writing skill or content knowledge targeted by an assessment: conventions (punctuation, capitalization, spelling), usage (grammar, diction), style (sentence structure and variety, diction), and organization (focus, elaboration, and support). (Rarely, writing applications skills are also assessed by multiple-choice items; one can easily see the difficulty of assessing any applied skill in this mode.)

There is some overlap (you see "diction" may fall into the usage camp or the style camp, for example, and sometimes grammar items fall into conventions, but basically, you might think of language items as mechanics and style/organization. Proofreading items then generally address mechanics, while editing and revision items may target either mechanics or style/organization, depending on the assessable skills.

Not all item writers can write language items. It's almost more of an editorly than a writerly undertaking, requiring a combination of specialized knowledge, persnicketiness, and an excellent grammar handbook. Some language items produced by unqualified writers are incomprehensible.

Sometimes these slip through the quality control cracks because language items are difficult to review because one must read very closely in order to identify that the error is an error, that the correct response is indeed correct, that the error is the type of error indicated by the standard/objective, and that there are no errors other than those intended. When we read, our brains automatically correct much of the error that we see, even when trained to do otherwise, so to review an item with intentional error that may also contain unintentional error is asking a lot. It's overwhelming, especially if you add in the editing passage, which means you have to do a lot of checking back and forth. Not to mention that a thorough knowledge of English usage and grammar is a rare commodity these days.

I have strong opinions about the content and construction of language items, to wit:
1. A language item should include only one type of error. A spelling item should not be contaminated by punctuation errors.
2. Each wrong answer choice should contain only one error.
3. The errors in the item must be the kinds of errors that students at the targeted grade level would reasonably make.
4. The errors should be obvious to the student who possesses the skill or content knowledge being assessed.
5. Trivial, why-bother sentences (or passages) should not be used to assess language skills. Language items should use actual facts in the stimulus sentences and paragraphs, rather than the easy but lame sentence.

Here is an example of an item with a trivial stimulus sentence:
Read the sentence.
 I haven't seen ______ since December.
Which pronoun should be used in the sentence?
A him *   B she    C they    D we

Here is an example of a language item based in fact:
Read the sentence.
 Langston Hughes had been writing poetry for years before Vachel Lindsay helped ____ publish his work.
 Which pronoun should be used in the sentence?
A he     B him *    C them    D they


I've written language items based on marine biology, space exploration, phenotypic plasticity, you name it. What is interesting in the world is a lot. There's no need to write about the purely meaningless.


UPDATE: Fixed some formatting with the MC items. Those are kind of tricky with the line breaks and indents.


UPDATE: Oh! I forgot to mention a recent dethspicable deplorable practice, that of using sentences from previously published (usually classic literature) as the stimulus for language items, either containing (newly imposed) embedded error or offering students options for improvements to the original (classic) writing.


Your mind is probably as boggled as mine was when first I came across this nastiness in the woodshed unspeakable horror unsound practice.


How might this work, you ask? Not well, as illustrated in the following examples.


Here is an example of the addition of error to a line from classic literature:



Read the sentence from Leo Tolstoy's Anna Karenina. 
Everyone thinks of changing the world, but no one thinks of changing themselves.
Which word would best replace the underlined word in the sentence? 
A himself*   B myself   C ourselves   D Leave as is.  


Here is an example of an invitation to students to improve upon a line from classic literature:


Read the sentence from Leo Tolstoy's Anna Karenina. 
Everyone thinks of changing the world, but no one thinks of changing himself.
To avoid repetition, which word would best replace the underlined word in the sentence? 
A altering*   B becoming   C improving   D renovating   
Why would anyone perform such sacrilege and blasphemy do such a thing? From the best of intentions.


An item writer, bored with the trivial, why-bother stimulus sentences, feels inspired to use sentences culled from the books she loves, sentences that are themselves little masterpieces of beauty, wit, and style. Won't this be good for the students? And there is no one to stop her, as these sentences are stolen borrowed from works now in the public domain.


Good intentions, road to you-know-where, cross-referenced to the law of unintended consequences.