7%

There was some point in junior high school when I stopped understanding math. I kept attending class, did the homework, and got good grades. In high school, I took algebra and geometry and then scored well enough on the PSAT, SAT, and ACT to have been recruited by colleges thousands of miles away from my home in the Appalachia of the West. What the.

Except for geometry (which I loved, I know not why), I did not understand a bit of anything having to do with numbers (as previously discussed here). How did I continue to do okay at something I did not at all understand?

Maybe my happiest moment in college was when I found out I never had to take another math class again ever, not ever, never. Years passed. I graduated. More years passed. I had a bunch of bad jobs, from hostess at Denny's to secretary at an auto repair shop (previously discussed here) to scheduler at a home health agency (where my biggest responsibility was bringing my supervisor a cup of coffee from the stand on the corner and then sitting in her office and listening to her talk about how the divorce was going). I never needed to know more math than what I was pretty competent with, i.e., adding, subtracting, multiplying, dividing, and finding percentages. Whew.

Then after three years of working in the criminal justice system, I decided that crime-fighting was not for me, and decided to return to my One True Love: English. Which would mean grad school, which would mean taking the GRE. I wasn't worried; I'd always kind of liked taking standardized tests, probably because it gave me a chance to do my favorite thing in the world: sit in a corner and read with no one talking to me. That the reading material wasn't always of the finest didn't trouble me. Like gutter winos who drank Night Train, I'd read whatever was available. (Still do. Yesterday while waiting for my daughter at the orthodontist's, I read Alaska Magazine, FORTUNE, and some other rich people magazine.)

So the first time I took the GRE, I did as well as one might expect on the verbal reasoning and analytical writing, and about as poorly as anyone could possibly do on the quantitative reasoning. I was no longer able to pass as someone with a minimally adequate understanding of math. I did so poorly that when I took the GRE a second time, I bubbled randomly for the quantitative reasoning and improved my score by 7%. I don't mean to mislead anyone; this made no significant improvement. If there had been a cut score for far below proficient, that is where my score would comfortably have settled like a little toad in a pond.

I was thinking about this because my daughters' CA STAR test scores came in the mail yesterday. And because I read this, about a grown man who submits to taking the SAT.

As a side note, I'd like to say that one might think this math handicap extends to data analysis, but it don't. I love data. Love it. I love the patterns--sometimes there is even a narrative.

I was reviewing longitudinal test result data for a high school and saw some patterns that might tell a story: a strong majority of incoming freshmen scored in the advanced category, but that there was a steep downward trajectory, with about half as many grade 11 students scoring so well. I have more investigating to do to be able to draw any meaningful conclusions--is this typical of all high school students in the district, state, country, or is this just this school? Could the difference partly be explained by a large influx of lower-performing students at grade 11? What other factors might influence these results?

These might be numbers, but there is a narrative, there are characters, there's a plot with conflict, action, and, one hopes, resolution.

Poppycock, Folderal, Nonsense

. . . in the immortal words of Todd Farley.

About a week ago, someone sent me a link to an Op-Ed piece in the New York Times by Todd Farley, author of Making the Grades: My Misadventures in the Standardized Testing Industry.

Farley's experiences aren't unique. Like Farley, I am a writer who sort of fell into the test publishing industry by accident. Like Farley, I stayed in the industry long after I thought I would have gone on to what I thought would be my real career of writing novels or screenplays or something, anything.

Both of us started our careers in hand-scoring, so hand-scoring is what I will talk about, specifically the hand-scoring of open-ended test questions. Multiple-choice questions are simple to score, because there is only one correct answer. All multiple-choice test questions are machine-scored. The answer sheets or test booklets are scanned, the answer choices verified by machine, and the scores are then computer-generated. Sometimes there are mistakes in the programming that must be corrected, for example, the correct answer to a given question was actually C but was identified somewhere along the line as A. Sometimes there are mistakes with a student's name or identification number that lead to a mistaken score. Sometimes--and this happened with my daughter's third-grade California STAR testbook--the testbook or answer sheet has juice spilled all over it, and so a false score may be generated. Where humans are involved, there will be some error somewhere, it is unavoidable, let us simply endeavor to put checks in place to catch the errors and processes to correct them.

The scoring of open-ended questions is a horse of a whole nother color. By its nature, there must be some subjectivity. In support of standardization are an array of tools that include a scoring guide or rubric, sample student responses at each score-point-level, and anchor papers and rangefinders. A rubric lists the characteristics of the response at each score point, a sample response gives an example of what kind of response is expected, an anchor is a student response that embodies the score point level, and rangefinders show what may be expected at the high, middle, and low ends of the spectrum within a score point level.

It sounds like a complicated process, and it is. And it's not without its ridiculous moments. And I have to say that though I found much about handscoring interesting, the work itself was tedious and the routine unbearable. But it's not the Orwellian circus of nonsense Farley describes. Or maybe it is at the company where Farley worked; it wasn't at CTB McGraw-Hill when I worked in hand-scoring there.

I am only about a quarter into the book, so maybe there will be some sort of Aristotelian discovery on Farley's part. At this point, he sounds like one of the disgruntled hand-scorers, and there were some of those, people who just never got it, never were able to internalize the scoring criteria and constraints, the ones whose scores had to be checked and re-checked so often that eventually they were let go. He says that he failed to qualify as a scorer for a writing test, which does make one wonder whether this type of work simply was not a good fit for him. Not that I can vouch for what happened at Pearson, as I've not worked there.

I will also say that--although I do not at all see myself as a flag-waver for the test publishing industry, and that I have my own strong feelings about the mis-use of tests and what seems to me to be an abuse of tests and how they are used and what they symbolize and how the data are manipulated--sitting in the mocking judgment seat is generally easy to do. I have plenty of ridiculous stories of my own. We humans are ridiculous, it's in our nature, and thank God that we are, it makes the world so much more entertaining.

And this book is just that--entertainment, a joke that is masked as an indictment of the industry. For myself, I'd be a lot more interested in a thoughtful exploration of the subject, one that takes into account the need for measurement in teaching, and the demand for standardization (because that seems to be the only way to ensure any kind of fairness or equity), and how we could possibly balance these kinds of standardized measurements with classroom performance and evaluations from teachers.

CORRECTION: I mean "Folderol." Geez. And to think I won first place in the 8th grade spelling bee. What did I tell you? Human error.