How to Get the Best from Item Writers

Many years ago, I was a development manager at a Great Big Huge Test Publishing Company. I've already told the story of how I began as a temp employee in hand-scoring, as so many recruits to the test publishing industry do. Armed with my book-learnin' and a new but hardly marketable M.A. in English, emphasis in creative writing, I was thrilled to get a job that paid slightly more than $10 an hour, a job that had to do with words and writing. Yay me, illustrating the joy of low expectations.

When the Great Big Huge Test Publishing Company was awarded what was then considered a big statewide assessment contract (back in the days when we tested at grades 3, 6, 8, and 10, or grades thereabouts), I was plucked from hand-scoring, handed the title of associate editor and deposited in a cubicle in a cavernous upstairs honeycomb which cubicle I shared with another associate editor who'd also come from hand-scoring. Within 5 years, I'd gone from the windowless cubicle of associate editor to content editor to supervisor to program manager to the window office of development manager. You can probably guess at my success as a manager, given I had no training and little experience in management. Oh, if only I had read Bringing Out the Best in People: How to Apply the Astonishing Power of Positive Reinforcement.

Which I have since read, and which principles I endeavor to apply when I'm called to supervise others, and to the effectiveness of which I can attest. We do what we know; when we know better, we do better.

It may be tempting, when we consider the plummeting quality of what we see on test materials, to blame the writers. But as this video from ETS reminds us, the writer is only one of many contributors.

Getting the best from item writers has to start long before an editor sends out that Are you available? email. The foundation of a project must be sound; there must be a blueprint and prototypes; there must be a clear vision of what the product is intended to look like, how it is intended to perform, what skills/knowledge it is intended to measure, and how it should measure those skills/knowledge.

These decisions should not be left to the item writer; few item writers are equipped to make such decisions. In the past, item writers worked in-house, or were mostly former employees of test publishing companies, and so were at least minimally conversant with principles governing the design and construction of assessments. Sometimes item writers were corralled to help assemble the tests and were given Xeroxed sheets containing lists of item numbers and associated data. That is no longer the case. I don't know of any test publishing company who maintains a staff of in-house item writers. Today test publishing companies commonly hire item writers who have never worked for test publishing companies and who have little experience writing items for high-stakes assessment (they may have written for curriculum and textbooks, if they have any experience at all). They may not have any classroom experience; they may not even have kids, and so the world of education--the real world of education and of what kids really are able to know and do at a given grade level--is a mystery to them. Or they develop their own ideas about what K-12 students know and can do, ideas that are as inaccurate as they are ambitious and inflated. (This is through no fault of their own, but the remedy is simple: volunteer in the classroom. Go to a school and offer to spend an hour a week in a classroom.)

Even if item writers were equipped, they shouldn't make decisions which should rightfully be made at a much higher level, by folks with greater knowledge, experience, and authority. Such decisions take time. There must be time to consider, reflect, think about it in the shower and in the car, time to return to one's colleagues and say Well, what if and how will it work if. The what-ifs must be given time to rise to the surface.

Rushing inevitably creates chaos. Whatever writers produce under slippery circumstances--when the expectations are not specified-- will fail to meet those unspecified expectations.

Assuming, however, that the big decisions have been made, and that the writers have been provided with everything they need to do a good (or excellent) job, what else can companies do to get the best from writers?

1. Take care of all housekeeping details upfront. Provide the writer with written information about the scope of work, schedule, deadlines, pay rates, and points of contact. Preferably all in one email message. Send the contract and the W-9. Tell the writer whom to invoice and how. Remove possible sources of worry. Worry is destructive to creativity and productivity. 
2. Provide training. The training should be as brief as possible, and should be conducted at the commencement of the project. A training that is offered a month before writing begins is useless, because writers will have forgotten the information they learned. Materials for the training should be emailed in advance. The writers should be told whom to call if they have questions.
3. There should be a dedicated content lead available to respond to writers' questions and to provide timely guidance throughout the course of the project.
4. Give writers the chance to do it right. The content lead's ducks must be lined up and ready to waddle. There must be a clear style to follow, preferences to comply with, and so on. The directions and feedback should be clear. To be effective, feedback must be immediate. Feedback must have the purpose of informing work in progress. Consider how disheartening it is to submit 50 items and then be told that there is now a new requirement, please revise those items accordingly and resubmit.
5. Allow the writers to work as they work best. More and more companies are requiring writers to input items directly into an online authoring system. While some of these are better than others, all add time and effort on the part of the writer, thus siphoning off energy better spent on item development. For each project, writers must learn how to use a new system; they might finish the project before they become proficient. Then it's off to a new system. I often decline opportunities to work in authoring systems, because I find the levels of clickage annoying--seconds add up to minutes add up to hours over the course of a year, hours I would much rather have spent reading or looking out the window or talking to my daughters or whatever else.
6. Let the writers do the work they do best. Writers write. Now that companies are operating on principles of leanness akin to corporate anorexia, companies are expecting writers to take on the work that used to be the province of content editors and desk-top publishers. With no increase in pay and no increase in time allotted to do the work.
7. Give writers enough space to write. Some assignments are so rigid and exacting, with so many criteria of so many types, that they become impossible.
8. Allow the writers to contribute their unique knowledge, experience, and skills. Writers work for all the educational assessment, test preparation, and curriculum publishers. They have access to a depth and breadth of knowledge about what's happening in educational publishing that is denied to the folks whose only job in educational publishing has been to work at the one company at which they are currently employed. Being open to the possibility that the writers know something and giving the writers freedom beyond the stricture This is how we do it will only serve the company and ultimately, the kids.
9. Be a human and let the writer be a human. We are none of us robots. We all have strengths and weaknesses. We have the skills we shine at and the skills we don't. This is normal and the nature of being human; it's not a flaw unique to a particular writer if she has trouble juggling multiple spreadsheets (not to name any names, me). No one in this world is capable of doing everything perfectly; no one is guilty of never making a mistake. The industry used to understand that; the protocol for test publishing included many rounds of editorial review prior to submitting materials to proofreading, and then to QA. 

If these principles were applied, quality would improve.

That's all I got for today. I'm off to Valencia, to the Cal Arts campus, to visit my daughter, Twin A ("A" being the initial written on the knitted cap the nurses placed on her head after her birth) who is a creative writer in the California State Summer School for the Arts program. 

What I'm reading: I finished As I Lay Dying. I love Faulkner. It always takes me at least half the book to marshall my resources to focus on his writing, I find it so challenging, but once I'm in, I'm there. I have a novel by Andre Brinks next, I think.

Whale in a Bathtub

Does anyone remember Helen Palmer's A Fish Out of Water, one of the classics from my childhood? The plot is simple, but compelling: A boy brings home a fish, overfeeds it, and the fish gets bigger and bigger and bigger until it is immense. I won't spoil the ending for you.

The need for growth management is one of the main points of what I've heard and read so far in the Coursera business class. Every owner of a successful business should carefully consider growth management. I'm thinking about small businesses now, like mine, but we've previously discussed problems associated with a failure to plan for and manage growth in larger companies. Quality is what seems to go out the window first when growth exceeds capacity.

This is not exclusive to educational publishing, or even publishing in general: it's a universal principle, as previously applied to dentistry.

Now I'd like to apply it to orthodontics. What up with the teeth? I have teen-agers, one of whom has recently been freed from braces, the other of whom is still wired up. They are twins, as you know, so why is one wired and the other free? The orthodontist's failure to manage the growth of his practice.

Like Dr. Bad Dentist, Dr. Too Successful Orthodontist came highly recommended. By three different people, one being my current dentist and another being an orthodontist in another state who went to school with Dr. Too Successful. During our consultation, he was friendly, explained everything clearly, answered all our questions, and seemed highly attentive.

Once we had signed on for a course of treatment for both my daughters, our experience turned southward. Dr. Too Successful's practice is booming. This means for a negative client experience, characterized by:

• a loud, crowded waiting room
• waits of up to and exceeding an hour for scheduled appointments
• difficulty in making appointments that will accommodate one's own schedule
• impossibility in rescheduling appointments
• technicians who are hurried and under stress, which doesn't really bring out the best in anybody
• necessity of having to see Dr. Too Successful's junior partner orthodontist
• a breakdown in communication between the client, technician, and doctor, resulting in a six month extension of treatment

The latter being what happened with us. That is, we were concerned about how my daughter's teeth were slanting, I mentioned it to the technician, she said that the junior partner doctor was aware of it, and I assumed he knew what he was doing. Three months later, when treatment was due to conclude, the junior partner doctor said that whoops, some girls look really pretty with teeth that slant outwards, it makes their lips look full, some people really like that. . .but if we wanted, we could have 4 teeth extracted and start all over. That was six months ago, my daughter is still in braces, and poor thing, she may be wearing them until she is thirty.

Regardless, the main point is that Dr. Too Successful has completely lost control of his business. Soon after my conversation with the junior partner doctor, I called Dr. Too Successful to let him know that I'd experienced an erosion of trust. He said he appreciated my call and we talked about what happened. I explained that I didn't fault his office for making a mistake, but what bothered me was that the mistake was preventable, being as it was a direct result of either the technician not telling the junior partner doctor about my concerns, or of the junior partner doctor simply not paying attention.

The call concluded with Dr. Too Successful assuring me that he was dedicated to regaining my trust. And so he had seemed to be, for the next few appointments, but then his attention was captured by other clients and more pressing demands, and our experience as clients is once again suffering.

Would I ever recommend Dr. Too Successful to anyone? No, on the contrary, I would tell everyone to flee as fast as their feet will carry them. I don't like saying this, because Dr. Too Successful is likable and seems committed to doing good work. And yet, by neglecting to control the growth of his practice and by forgetting to consider his clients' experience, Dr. Too Successful caused my daughter unnecessary pain and me unnecessary inconvenience. And he has had a negative effect on my business, in that I've had to take additional time away from my work to keep taking my daughter to orthodontist appointments, appointments for which I always have to wait at least 20 minutes, and often have to wait an hour. 

Here are the questions I ask myself:

• What is my typical client's experience?
• Is there any negative aspect to my client's experience of Inkspot?
• How can I improve my client's experience?

One of the little mottoes of the business course is that one has to love the client more than one loves the product. It's worth thinking about.

UPDATE: Lest anyone think I am casting stones from a glass house, I'd like to add this bit of irony. Last week, I had to call and reschedule an appointment at the last minute because my daughters had a school obligation arise suddenly that could not be missed. I was told the next available appointment was toward the end of February. The day and time offered conflicted with cello lessons, and I was offered an appointment in mid-March. By this time, feeling frustrated with having to find a time that fit my daughter's schedule, my schedule, and the straitjacketed schedule of the orthodontist, I said that I really needed the office to accommodate me and be more flexible, and here I mentioned that my daughter's treatment had only been extended to this point as a result of mistakes made in their office. I was given an appointment for yesterday.

We went to the office, and even though everyone was courteous, it was strained, formal courtesy. Something was amiss, something more than what had passed between the receptionist and me on the phone. When the doctor examined my daughter's teeth, he spoke exclusively to her, as if I weren't there.

At the end of the appointment, I opened my planner to make our next appointment and saw that WE HAD ARRIVED AN HOUR LATE. That's right. In my mind, our appointment was at 2:30, because that was the time of the original appointment, but we should have been there at 1:30 and NO ONE HAD SAID A WORD. Though their body language and tone said plenty.

I was so astounded that I didn't know what to do, but immediately upon arriving home, I called the office to offer abject apologies. I apologized about ten different ways, and I sure hope that my apologies made their way to the technicians and the doctor.

And yet? You understand that the doctor has never made that kind of apology to me for their mistake. I was an hour late for an appointment, but their mistake means that my daughter will remain in braces for what looks like a year after the original estimated end of treatment. But I guess some people find it much more difficult to apologize than others.

Worry Plate

Title courtesy of Julie Allinson, founder of Eyebobs, purveyor of fashionable eyewear, from an interview provided as part of the content for this online business course I'm taking.

What's on the worry plate? Let's just say that mistakes were made. To err may be human, but nobody likes to be reminded just how human one's service provider is.

For a dozen years, I went to a dentist who had been recommended to me. He was known for being the reassuring, kindly dentist who went easy on anxious patients. His teeth gleamed, he wore a huge gold watch and always drove a newer model of European luxury car, he was hale and hearty and volubly conservative in his politics (he talked a lot about politics while performing his work; while I didn't share his views, I was loathe to engage him in debate, partly because who wants to argue with someone wielding sharp metal instruments near one's face, and partly because it was impossible to talk while holding my jaws open), and he spent a great deal of the winter skiing or talking about skiing. (Summer was dedicated to tennis.) Already in his sixties when I became his patient, the dentist died several years ago, after I had moved away.

Two years ago, I learned that this dentist had been a bad dentist. Expensive and unpleasant consequences followed this discovery. With the help of my current dentist, I tried to submit a claim for Dr. Bad Dentist to pay for the rework, but then I learned that Dr. Bad Dentist had departed this earthly realm, and that his malpractice insurance ceased covering claims a year after his departure.

So I asked my current dentist about how a consumer can tell that a dentist is good. Dentistry is highly specialized; what average consumer would be able to distinguish shoddy dental work from excellent dental work? How could I have possibly known that the cause of some of those trips to Dr. Bad Dentist was actually Dr. Bad Dentist? We talked about getting a second opinion, but what if the dentist offering the second opinion was Dr. Worse Dentist? Or what if the first dentist was Dr. Good Dentist and the second opinion was offered by Dr. Terrible No Good Dentist? There's really no way to know. My current dentist just shook his head and said you have to find someone you can trust. And I said again to him, Well, and how do I know I can trust someone?

I was thinking about how sleek and prosperous and jovial and reassuring Dr. Bad Dentist was, never failing in his good cheer, and yet--he did some damage is what I'm saying. 

My current dentist showed me some X-rays of what a filling should look like. He explained why it should look that way, and told me the principle underlying the practice. It all made sense to me. Then he showed me an X-ray of a filling of Dr. Bad Dentist's doing. He showed me the flaws and explained why they were flaws. This show-and-tell went a long way with me.

The story of Dr. Bad Dentist is just an illustration. I'm thinking about mistakes I've made and mistakes my subcontractors have made--ones that I caught before I delivered them and ones that I did not.

The mistakes are one thing--but one has to work with people whom one trusts to do their best work, to know what that best should be (and some people truly don't know and truly are unable to distinguish the good from the bad, but that is a different problem), and to fix mistakes when they make them.

From the consumer angle: ask questions. Ask a lot of them. Keep asking until you understand enough to be able to tell if the words make sense. More on this later.