How to Get the Best from Item Writers

Many years ago, I was a development manager at a Great Big Huge Test Publishing Company. I've already told the story of how I began as a temp employee in hand-scoring, as so many recruits to the test publishing industry do. Armed with my book-learnin' and a new but hardly marketable M.A. in English, emphasis in creative writing, I was thrilled to get a job that paid slightly more than $10 an hour, a job that had to do with words and writing. Yay me, illustrating the joy of low expectations.

When the Great Big Huge Test Publishing Company was awarded what was then considered a big statewide assessment contract (back in the days when we tested at grades 3, 6, 8, and 10, or grades thereabouts), I was plucked from hand-scoring, handed the title of associate editor and deposited in a cubicle in a cavernous upstairs honeycomb which cubicle I shared with another associate editor who'd also come from hand-scoring. Within 5 years, I'd gone from the windowless cubicle of associate editor to content editor to supervisor to program manager to the window office of development manager. You can probably guess at my success as a manager, given I had no training and little experience in management. Oh, if only I had read Bringing Out the Best in People: How to Apply the Astonishing Power of Positive Reinforcement.

Which I have since read, and which principles I endeavor to apply when I'm called to supervise others, and to the effectiveness of which I can attest. We do what we know; when we know better, we do better.

It may be tempting, when we consider the plummeting quality of what we see on test materials, to blame the writers. But as this video from ETS reminds us, the writer is only one of many contributors.

Getting the best from item writers has to start long before an editor sends out that Are you available? email. The foundation of a project must be sound; there must be a blueprint and prototypes; there must be a clear vision of what the product is intended to look like, how it is intended to perform, what skills/knowledge it is intended to measure, and how it should measure those skills/knowledge.

These decisions should not be left to the item writer; few item writers are equipped to make such decisions. In the past, item writers worked in-house, or were mostly former employees of test publishing companies, and so were at least minimally conversant with principles governing the design and construction of assessments. Sometimes item writers were corralled to help assemble the tests and were given Xeroxed sheets containing lists of item numbers and associated data. That is no longer the case. I don't know of any test publishing company who maintains a staff of in-house item writers. Today test publishing companies commonly hire item writers who have never worked for test publishing companies and who have little experience writing items for high-stakes assessment (they may have written for curriculum and textbooks, if they have any experience at all). They may not have any classroom experience; they may not even have kids, and so the world of education--the real world of education and of what kids really are able to know and do at a given grade level--is a mystery to them. Or they develop their own ideas about what K-12 students know and can do, ideas that are as inaccurate as they are ambitious and inflated. (This is through no fault of their own, but the remedy is simple: volunteer in the classroom. Go to a school and offer to spend an hour a week in a classroom.)

Even if item writers were equipped, they shouldn't make decisions which should rightfully be made at a much higher level, by folks with greater knowledge, experience, and authority. Such decisions take time. There must be time to consider, reflect, think about it in the shower and in the car, time to return to one's colleagues and say Well, what if and how will it work if. The what-ifs must be given time to rise to the surface.

Rushing inevitably creates chaos. Whatever writers produce under slippery circumstances--when the expectations are not specified-- will fail to meet those unspecified expectations.

Assuming, however, that the big decisions have been made, and that the writers have been provided with everything they need to do a good (or excellent) job, what else can companies do to get the best from writers?

1. Take care of all housekeeping details upfront. Provide the writer with written information about the scope of work, schedule, deadlines, pay rates, and points of contact. Preferably all in one email message. Send the contract and the W-9. Tell the writer whom to invoice and how. Remove possible sources of worry. Worry is destructive to creativity and productivity. 
2. Provide training. The training should be as brief as possible, and should be conducted at the commencement of the project. A training that is offered a month before writing begins is useless, because writers will have forgotten the information they learned. Materials for the training should be emailed in advance. The writers should be told whom to call if they have questions.
3. There should be a dedicated content lead available to respond to writers' questions and to provide timely guidance throughout the course of the project.
4. Give writers the chance to do it right. The content lead's ducks must be lined up and ready to waddle. There must be a clear style to follow, preferences to comply with, and so on. The directions and feedback should be clear. To be effective, feedback must be immediate. Feedback must have the purpose of informing work in progress. Consider how disheartening it is to submit 50 items and then be told that there is now a new requirement, please revise those items accordingly and resubmit.
5. Allow the writers to work as they work best. More and more companies are requiring writers to input items directly into an online authoring system. While some of these are better than others, all add time and effort on the part of the writer, thus siphoning off energy better spent on item development. For each project, writers must learn how to use a new system; they might finish the project before they become proficient. Then it's off to a new system. I often decline opportunities to work in authoring systems, because I find the levels of clickage annoying--seconds add up to minutes add up to hours over the course of a year, hours I would much rather have spent reading or looking out the window or talking to my daughters or whatever else.
6. Let the writers do the work they do best. Writers write. Now that companies are operating on principles of leanness akin to corporate anorexia, companies are expecting writers to take on the work that used to be the province of content editors and desk-top publishers. With no increase in pay and no increase in time allotted to do the work.
7. Give writers enough space to write. Some assignments are so rigid and exacting, with so many criteria of so many types, that they become impossible.
8. Allow the writers to contribute their unique knowledge, experience, and skills. Writers work for all the educational assessment, test preparation, and curriculum publishers. They have access to a depth and breadth of knowledge about what's happening in educational publishing that is denied to the folks whose only job in educational publishing has been to work at the one company at which they are currently employed. Being open to the possibility that the writers know something and giving the writers freedom beyond the stricture This is how we do it will only serve the company and ultimately, the kids.
9. Be a human and let the writer be a human. We are none of us robots. We all have strengths and weaknesses. We have the skills we shine at and the skills we don't. This is normal and the nature of being human; it's not a flaw unique to a particular writer if she has trouble juggling multiple spreadsheets (not to name any names, me). No one in this world is capable of doing everything perfectly; no one is guilty of never making a mistake. The industry used to understand that; the protocol for test publishing included many rounds of editorial review prior to submitting materials to proofreading, and then to QA. 

If these principles were applied, quality would improve.

That's all I got for today. I'm off to Valencia, to the Cal Arts campus, to visit my daughter, Twin A ("A" being the initial written on the knitted cap the nurses placed on her head after her birth) who is a creative writer in the California State Summer School for the Arts program. 

What I'm reading: I finished As I Lay Dying. I love Faulkner. It always takes me at least half the book to marshall my resources to focus on his writing, I find it so challenging, but once I'm in, I'm there. I have a novel by Andre Brinks next, I think.