Wednesday, October 7, 2009

An Angry Little Toot from a Lone Brave Whistle-Blower

The books I am reading right now are: Love and Will by Rollo May, Aretha Franklin's autobiography Aretha: From These Roots, and Daniel Goleman's Vital Lies, Simple Truths: The Psychology of Self-Deception. And I just finished Making the Grades: My Misadventures in the Standardized Testing Industry.

There may not, at first glance, appear to be any common ground (in the immortal words of Rev. Jesse Jackson) among these, but I will argue that as humans, we take ourselves wherever we go, and in so doing, we drag along the burdens of either consciousness or self-deception. As we choose. Sometimes one, sometimes the other, as best meets our needs and best fits our capacities at the time, the question of existence often being the same as that faced by Oedipus, (as Rollo May says): how much self-awareness can a human being bear? One hopes--I hope--that one may travel an upward trajectory in which one increases one's awareness of oneself and of the world, a trajectory that leads to some higher plane in which we can learn to live with truth.

We take ourselves to work, where we sometimes must walk a tightrope between our values and the need to earn a living. Most of us must make some compromises, must sell ourselves in some way. Some compromises are small and meaningless, but others may put our very integrity at stake. This is the real story of Farley's book.

Though the book is purportedly about the testing and test publishing industries, it is just as much about Farley, who presents himself as a whistle-blower, though one might be forgiven for the mean-spirited thought that Farley sure did take his time in finding the whistle, being as he kept on collecting a paycheck from The Great Satans for many years. And that perhaps this delay lends a bit of tarnish to his credibility.

It must also be said that Farley does not appear to best advantage when he writes about how he copied other workers' scores to get out of doing the work himself, or about spending most of his workday surfing the Internet, or about his hand-rubbing glee in charging exorbitant fees as a consultant (though maybe my pointing out the latter is evidence of envy on my part, as I cannot help but wonder how he managed to pull this off, as I have been a consultant in this industry for 8 or 9 years, and though I do support my little family, our style of living cannot be described as high off even a tiny hog). (Not to mention the subtle sexism in Farley's thinking that is revealed in his writing. Look at how his view of women is first invariably filtered through the lens of whether or not he finds them attractive. In his mind, women--no matter how accomplished or intelligent--are reduced to decorative objects because of course, a woman's main value has to do with whether you enjoy gaping at her. Then consider the adjectives and nouns he uses when writing about women. He says he traveled with "a gaggle" of women. Oh, please. Yes, we of the feminine persuasion are all just clacking geese, you know how ladies love to gab. Sigh.)

Reading Farley's book raises as many questions about him as it does about the industry he intends to expose. If it were so chock-full of despicable practices, why did he remain there for 15 years? How did making such a sacrifice of his own values and beliefs affect him? What exactly was going on in his mind as he participated in these ethics violations? While working in that industry, what efforts did he make for reform?

It seems that Farley wants to rail against this industry-wide malfeasance without taking any responsibility for his own role in it, but as it do say in the Bible, one cannot touch pitch and not be defiled.

This is a dilemma in which you might say I have a deep and abiding interest. The spirit of full disclosure compels me to state that I worked at CTB McGraw-Hill from 1993 to 2001, since which time I have been a content development consultant for a variety of test publishers, school districts, and one state department of education. Having worked with most of the major test publishing companies, I can say that I have seen a good share of corporate culture, and the more I see, the more I am glad I work for myself. Anyone who has ever worked in a corporate setting is probably familiar with at least some of the horrors Farley describes. People who are like the walking dead, who are so eccentric and odd-mannered as to seem unemployable, catch-22 mandates handed down from upper-level management, meetings that seem designed to showcase pomposity and vanity and futility.

But the shenanigans of wrong-doing in hand-scoring that Farley reveals, the behind-the-scenes falsification of scores, the pressure to score in one direction or another as the wind from on high changes, demands from psychometricians to increase the number of scores at a given grade point level, hand-scorers who were the dregs of society--these I did not see. Pressure to work faster, for higher productivity, yes. Unreasonable demands, yes. Obsequious sucking up to district or state officials, yes, yes, yes. Lots of co-workers with their little quirks, oh, yes.

As there is in the world, there is much that could and should be improved in this industry. On all sides, and probably in every department in every test publishing company. If Farley says that what he describes in the book was his experience, then that was his experience, I will not dispute that, though my own experience has been different. I agree completely that tests today are being used for purposes for which they should not be used. I agree completely that there is more to learning than can be measured by a paper and pencil test (or a keyboard test). I agree completely that one of the unintended consequences of the No Child Left Behind legislation has been the unleashing of unprincipled money-sniffing dogs into the industry (no offense to literal dogs), and the muck they try to pass off as genuine content--well! I have seen some awful terrible bad no-good things, is what I am saying.

And yet, testing is never going to disappear. Nor should it. The example I always give when a stranger tries to hold my feet to the fire is whether you would want to undergo an invasive medical procedure at the hands of a surgeon who had never submitted to (let alone achieved a passing grade from) any kind of examination as to his or her knowledge and skill. Let's face it, we don't even want to take our cars to mechanics who are not certified in some manner.

See, we can construct this evil villain testing empire, we can make paper dragon cut-outs all we want, but how does that effect real change? What about starting where we are? For myself, I find that much of my work has been coming more from curriculum the last few years, partly because these particular clients are just plain charming and nice to work with, but partly because, given the choice, I would rather be working on the side of remediation and intervention. Not that I have stopped my assessment work. However, if I ever do feel about it the way Farley did--if it stole from me my integrity--I hope that I would not hesitate to leave it behind. I can't really know what I would do unless I find myself in that situation. Some sleepless nights would result, I am sure. Rollo May says that fate plus guilt equals no rest for the wicked. (My paraphrase.)

Poppycock, Folderal, Nonsense

. . . in the immortal words of Todd Farley.

About a week ago, someone sent me a link to an Op-Ed piece in the New York Times by Todd Farley, author of Making the Grades: My Misadventures in the Standardized Testing Industry.

Farley's experiences aren't unique. Like Farley, I am a writer who sort of fell into the test publishing industry by accident. Like Farley, I stayed in the industry long after I thought I would have gone on to what I thought would be my real career of writing novels or screenplays or something, anything.

Both of us started our careers in hand-scoring, so hand-scoring is what I will talk about, specifically the hand-scoring of open-ended test questions. Multiple-choice questions are simple to score, because there is only one correct answer. All multiple-choice test questions are machine-scored. The answer sheets or test booklets are scanned, the answer choices verified by machine, and the scores are then computer-generated. Sometimes there are mistakes in the programming that must be corrected, for example, the correct answer to a given question was actually C but was identified somewhere along the line as A. Sometimes there are mistakes with a student's name or identification number that lead to a mistaken score. Sometimes--and this happened with my daughter's third-grade California STAR testbook--the testbook or answer sheet has juice spilled all over it, and so a false score may be generated. Where humans are involved, there will be some error somewhere, it is unavoidable, let us simply endeavor to put checks in place to catch the errors and processes to correct them.

The scoring of open-ended questions is a horse of a whole nother color. By its nature, there must be some subjectivity. In support of standardization are an array of tools that include a scoring guide or rubric, sample student responses at each score-point-level, and anchor papers and rangefinders. A rubric lists the characteristics of the response at each score point, a sample response gives an example of what kind of response is expected, an anchor is a student response that embodies the score point level, and rangefinders show what may be expected at the high, middle, and low ends of the spectrum within a score point level.

It sounds like a complicated process, and it is. And it's not without its ridiculous moments. And I have to say that though I found much about handscoring interesting, the work itself was tedious and the routine unbearable. But it's not the Orwellian circus of nonsense Farley describes. Or maybe it is at the company where Farley worked; it wasn't at CTB McGraw-Hill when I worked in hand-scoring there.

I am only about a quarter into the book, so maybe there will be some sort of Aristotelian discovery on Farley's part. At this point, he sounds like one of the disgruntled hand-scorers, and there were some of those, people who just never got it, never were able to internalize the scoring criteria and constraints, the ones whose scores had to be checked and re-checked so often that eventually they were let go. He says that he failed to qualify as a scorer for a writing test, which does make one wonder whether this type of work simply was not a good fit for him. Not that I can vouch for what happened at Pearson, as I've not worked there.

I will also say that--although I do not at all see myself as a flag-waver for the test publishing industry, and that I have my own strong feelings about the mis-use of tests and what seems to me to be an abuse of tests and how they are used and what they symbolize and how the data are manipulated--sitting in the mocking judgment seat is generally easy to do. I have plenty of ridiculous stories of my own. We humans are ridiculous, it's in our nature, and thank God that we are, it makes the world so much more entertaining.

And this book is just that--entertainment, a joke that is masked as an indictment of the industry. For myself, I'd be a lot more interested in a thoughtful exploration of the subject, one that takes into account the need for measurement in teaching, and the demand for standardization (because that seems to be the only way to ensure any kind of fairness or equity), and how we could possibly balance these kinds of standardized measurements with classroom performance and evaluations from teachers.

CORRECTION: I mean "Folderol." Geez. And to think I won first place in the 8th grade spelling bee. What did I tell you? Human error.