Gap

As obvious as the link is between the quality of content (we could also say validity) and item writing training, it remains a mystery to me how sound practices in item writing training have nearly become obsolete.


Steven M. Downing discusses this link in "Twelve Steps for Effective Test Development" from the Handbook of Test Development:

Yet knowing the principles of effective item writing is no guarantee of an item writer's ability to actually produce effective test questions. Knowing is not necessarily doing. Thus, one of the more important validity issues associated with test development concerns the selection and training of item writers . . . . The most essential characteristic of an effective item writer is content expertise. Writing ability is also a trait closely associated with the best and most creative item writers.
. . .
Effective item writers are trained, not born. Training of item writers is an important validity issue for test development. Without specific training, most novice item writers tend to create poor-quality, flawed, low-cognitive-level test questions that test unimportant or trivial content. Although item writers must be expert in their own disciplines, there is not reason to believe that their subject matter expertise generalizes to effective item writing expertise. Effective item writing is a unique skill and must be learned and practiced. For new item writers, it is often helpful and important to provide specific instruction using an item writer's guide, paired with a hands-on training workshop (Haladyna, 2004). As with all skill learning, feedback from expert item writers and peers is required. The instruction-practice-feedback-reinforcement loop is important for the effective development and maintenance of solid item writing skills. . . .

The best and most effective training, then, is to teach item-writing skills to content area experts, give them a guide for reference, let them practice, review and comment on their work, have them make revisions, and review and comment again until the items are satisfactory. Add a peer review step -- but only if the peers have a thorough understanding of the principles for developing sound items. Repeat as necessary.


In other news, April is National Poetry Month. I like the idea of carrying a poem in your pocket.