Measure What Can Be Measured

I'm in DC for a joyous occasion, and even so--even here, in the midst of the joy, even in the thrill of riding the Metro and the suspense of wondering if this be the ride during which I become irretrievably lost in the bowels of the city--I'm thinking about readability.

In assessment publications, readability, or the grade at which we might reasonably expect a student to be able to read a given text, is determined by one or more of these measures:

• a readability formula, such as Flesch-Kincaid, Lexile, or another
• the monitoring of vocabulary according to a graded word list, such as EDL Core Vocabularies (Steck Vaughn) or The Children's Writer's Word Book (Mojilner)
• content area expertise
• teacher review

Because of the Common Core Standards, one can no longer talk about readability without text complexity pulling up a chair to the table. Which is all to the good.

Here is an explanation of Lexile by Jason Turner at MetaMetrics (by posting this, I intend neither to promote nor disparage Lexile, simply to offer information). I'd like to clarify that when Turner identifies Lexile as a measure of text complexity, he means that it is a quantitative measure--there is no readability formula capable of providing a qualitative measure. 

This is such a vital point that I feel compelled to repeat that there is no readability formula capable of providing a qualitative measure. 

Readability formulae cannot interpret nor analyze meaning. No theme, motif, trope, metaphor, symbol--no beauty, no lyricism--can be interpreted nor analyzed by a readability formula, which greatly diminishes the likelihood that such a formula could provide an accurate measure of literary text. Readability formulae can count.

What is counted--word frequency, word length, sentence length--may vary from formula to formula. It doesn't seem to matter much what is counted; like the yellow, green, orange, and blue lines of the DC Metro trains that all pull up to L'Enfant Plaza, the readability formulae tend to arrive at the same conclusions: "No one of the quantitative measures performed significantly differently than the others in predicting student outcomes."

These formulae, therefore, are most useful in evaluating the suitability of informational text for students at a certain grade level, much less useful in evaluating the suitability of literary text, and not at all useful for evaluating the suitability of poetry or drama.

Here is an explanation published by the Council of Chief State School Officers and the National Governors' Association:

There will be exceptions to using quantitative measures to identify the grade band; sometimes qualitative considerations will trump quantitative measures in identifying the grade band of a text, particularly with narrative fiction in later grades. Research showed more disagreement among the quantitative measures when applied to narrative fiction in higher complexity bands than with informational text or texts in lower grade bands. Given this, preference should sometimes be given to qualitative measures when evaluating narrative fiction intended for students in grade 6 and above. For example, some widely used quantitative measures rate the Pulitzer Prize-winning novel Grapes of Wrath as appropriate for grades 2–3. This counterintuitive result emerges because works such as Grapes often express complex ideas or mature themes in relatively commonplace language (familiar words and simple syntax), especially in the form of dialogue that mimics everyday speech. Such quantitative exceptions for narrative fiction should be carefully considered, and exceptions should be rarely exercised with other kinds of text. It is critical that in every ELA classroom students have adequate practice with literary non-fiction that falls within the quantitative band for that grade level. To maintain overall comparability in expectations and exposure for students, the overwhelming majority of texts that students read in a given year should fall within the quantitative range for that band.

It seems clear, then, that for literary text we must rely on a second opinion. And yet, from what I see, reviewers (content editors at test publishing companies, panels of teachers convened to evaluate test materials) continue to rely primarily on quantitative formulae rather than qualitative considerations when deciding to accept or reject literary passages. Just as we can all agree that Grapes of Wrath is hardly a book for second- and third-graders, I think we can all agree that this is a mistake.

I understand why and how this mistake is made. Unless trained in the study of literature (which is and has been a bona fide field of study for centuries upon centuries), who feels qualified to make decisions about a literary work? And yet, in the immortal words of Galileo, here we must find some way to "make measurable what cannot be measured."

The good news is that there are many people who are so trained. Why not add their voices to the mix? This would surely help us remain true to the goal of creating the best fit possible between reader and text.



 
And now, I'd like to return to the beginning and wish Kia and Roy, the two most brilliant people I know, two who are blessed with every gift of intellect and heart, two who are not just brilliant writers but purely lovely human beings, every possible happiness. The Donne poem is from the ceremony.

THE GOOD-MORROW
by John Donne

I WONDER by my troth, what thou and I
Did, till we loved? were we not wean'd till then ?
But suck'd on country pleasures, childishly ?
Or snorted we in the Seven Sleepers' den ?
'Twas so ; but this, all pleasures fancies be;
If ever any beauty I did see,
Which I desired, and got, 'twas but a dream of thee.

And now good-morrow to our waking souls,
Which watch not one another out of fear;
For love all love of other sights controls,
And makes one little room an everywhere.
Let sea-discoverers to new worlds have gone;
Let maps to other, worlds on worlds have shown;
Let us possess one world; each hath one, and is one.

My face in thine eye, thine in mine appears,
And true plain hearts do in the faces rest;
Where can we find two better hemispheres 
 Without sharp north, without declining west?
Whatever dies, was not mix'd equally;
If our two loves be one, or thou and I
Love so alike that none can slacken, none can die.

Source:
Donne, John. Poems of John Donne. vol I. E. K. Chambers, ed. London: Lawrence & Bullen, 1896. 

NOTE: Just for fun, here are the Flesch-Kincaid measures for both the John Donne poem and this blog post.
Flesch-Kincaid measure for "The Good-Morrow": 2.1
Flesch-Kincaid measure for "What Can Be Measured": 12.0

NOTE THE SECOND: I listened to this bit with Martha Nussbaum on the value of the humanities and wanted to share.

What the World Needs Now

It may be unnecessary to say that in talking about what's wrong, I don't mean to pick on anybody in particular. Even if it is one person making specific mistakes, I tend to see this as a systemic ill, as previously discussed, rather than an opportunity to dogpile on an unwitting offender.

A systemic ill that is particularly pernicious in the test publishing industry. How can we expect kids to do well on tests when the tests themselves are riddled with error?

The safeguards are so simple:

1. provide adequate training (again, as previously discussed)

2. follow some kind of standard process for detecting and repairing error

3. use the information from #2 to supplement #1

These safeguards cost time and money, which explains why so many companies have gradually let them fall by the wayside. 

Training takes time and attention. I can't believe how fortunate I was to have received the training I did. In my first year as an associate editor, I probably had more training opportunities than editors now receive in a decade. It was a different era. The Great Big Huge Test Publishing Company at which I worked had an army of style editors, proofreaders, and QA personnel.

Over the last dozen years, I've seen companies completely dismantle their style editing departments and outsource the work to freelance editors. Now style editors (copy editors) are held in such low regard that if any Bright Young Thing with a talent for grammar and language conventions tells me she'd like to give this work a try, I steer her away from style editing and into content.  I hate doing it. I myself hold style editors in high esteem, maybe because often their job seems to be to keep me from making a fool of myself. What the world needs now--no disrespect intended to Dionne Warwick and Burt Bacharach--is more style editing.

Many companies have also dismantled their content development departments.

So there's no real continuity for item writers and content editors. They work on a project for one company and then skip to a project for another. The content development people who remain at the companies are overwhelmed and certainly don't have the time to provide training to freelancers; editors accept or reject items, which means they're likely to fix mistakes themselves, and so the item writer never hears of his mistakes and is free to repeat those mistakes forever in a state of unfortunate ignorant complacence.

Right now, I'm thinking of all this in terms of language conventions items, because I was reviewing a set this morning, but I'm sure this applies to all domains and content areas.

Right now, I offer some specific observations about language items:
1. When language items accompany an editing passage that contains embedded error, the content of each item must be mutually exclusive. A sentence from the passage should contain only one error, and should be used only for one item in a set.
2. "Syntax" has to do with grammatical rules; "diction" has to do with the writer's choice of words.
3. An item should clearly target only one type of error, and should not mix types of errors. For example, a punctuation item shouldn't include distractors that contain errors in verb tense.

More previously discussed here.

As an aside, it's interesting how you can learn a lot about a person by reviewing his or her work, and how this body of knowledge grows over time. It's like how my mechanic told me that he knows more about his customers from the condition of their cars than he does from talking with them. He can tell who drives erratically, who slams on the brakes, who grinds the gears, and he can tell who doesn't. Everything is everything. You can develop a similar personality profile simply by reviewing a writer's work.

I can't tell you how many times I've finished working on a manuscript, items or reading passage, don't make no difference, and felt a deep and tender affection for the writer because his or her work demonstrated care, attention, and conscientiousness. 

UPDATE: Forgot to mention. I started writing about all this because I saw this book mentioned in the Common Core Standards and immediately ordered it. 

Jobs, Money, and Love for All

Good grammar and spelling skills make your life better.

From Education Week, a teacher's blog post about an employer who screens job candidates to weed out those with poor English grammar skills:

In a fiery post for the Harvard Business Review, Wiens says he flat out won't hire people who are careless with grammar. And to ensure that no offenders slip through, both of his companies—Wiens is also the founder of the documentation-software maker Dozuki—have instituted mandatory grammar tests as part of the hiring process.

Before I read the source post by Kyle Wiens for Harvard Business Review, I had mixed feelings. Is this really necessary for candidates who aren't writers? I know so many smart, capable people who use--oh, let's call it "nonstandard grammar." They're not writers, and so I switch off the inner critic before I read their email messages.

But Wiens makes a compelling argument:

Grammar signifies more than just a person's ability to remember high school English. I've found that people who make fewer mistakes on a grammar test also make fewer mistakes when they are doing something completely unrelated to writing — like stocking shelves or labeling parts. 

In the same vein, programmers who pay attention to how they construct written language also tend to pay a lot more attention to how they code. You see, at its core, code is prose. Great programmers are more than just code monkeys; according to Stanford programming legend Donald Knuth they are "essayists who work with traditional aesthetic and literary forms." The point: programming should be easily understood by real human beings — not just computers. 

And just like good writing and good grammar, when it comes to programming, the devil's in the details. In fact, when it comes to my whole business, details are everything. 

He's right that details matter.

You can even put a price on details. According to a 2004 article in The New York Times, one spelling mistake can cost as much as $198:

Mr. Green once bought a box of gers for $2. They were gears for pocket watches, which he cleaned up and put back on the auction block with the right spelling. They sold for $200. 

An article published by BBC News in 2011 calculates the cumulative costs:

An online entrepreneur says that poor spelling is costing the UK millions of pounds in lost revenue for Internet businesses.

Wayne State University assistant professor Fred Vultee, who conducted a study sponsored by the American Copy Editors' Society, found that not only can readers distinguish edited from unedited text, but they notice and are troubled by errors (more here). Said Vultee, "Editing makes a difference. It's across the board, it's not imaginary, and it's reasonably big."

And if that weren't enough, errors in spelling and grammar can also inhibit romance: "In a survey of more than 5,000 Match.com members asking about the most common profile mistakes, 51% complained about profiles with poor spelling or grammar...."

Over the last almost twenty years, I've worked with writers whose English language skills range from impeccable to not. When I say "not," I don't mean to pick on anyone who makes an occasional typo or some other minor mistake. It's very difficult to proof one's own work; the brain insists on making the text appear as you know it should appear. I use "not" for those whose work displays not just a repeated pattern of error, but sometimes repeated patterns of different types of errors, even after those same types of errors have been pointed out to the writer previously.

Is it prejudice to expect that people whose trade is writing should have strong language skills? Maybe. For me, it's work avoidance; writers who make lots of mistakes cause me lots of extra work. Which means I can verify that mistakes have a cost, because if I need writers, my first calls are to the ones whose writing causes me the least work in terms of content and style.

I don't fault anyone for not knowing something; we all of us stumble on our ignorance sometimes. And yet, it works against us to cling to that ignorance once we become aware of it.

Toward the goal of jobs, money, and love for all, I offer the following resources:
Grammar & Style: Grammar Guides, Style Guides, APA, MLA--LibrarySpot.com
Grammar.net
HyperGrammar at the University of Ottawa
On Writing Well by William Zinsser
Pam Nelson: Grammar Guide
The Elements of Style, Fourth Edition by William Strunk, E.B. White, and Roger Angell
The Purdue Online Writing Lab
Top Ten Resources on Spelling and Word Study

Thinking

The train is rolling into the station on our big passage writing project. As my co-conspirator partner-in-crime colleague cautions, it's too soon to uncork the champagne. There is still work to do, of course, in the immortal words of the Isley Brothers.

We have two other projects we're working on, too, so one train arrives for the disembarkation while another arrives and the passengers load up. And yet, we foresee a time when we'll have completed what we're doing now. Then what? We're thinking. We're big on thinking. In order to think, I always need to go looking for raw material to chip and chop. In that search, I found some items of interest:


SchoolTube: School videos. Who knew.
National Novel Writing Month: Coming up. Sharpen your pencils. And your wits.
First Book: Providing books to children in need.
National Writing Project: Writing prompts, resources, articles about writing and teaching writing.
Best Ever Teen Novels: Vote. Though how anyone could vote for anything else with To Kill a Mockingbird on the list is beyond me.


That's all I got. I'm going back in.

The Habit of Industriousness

All knowledge is of itself of some value. There is nothing so minute or inconsiderable, that I would not rather know it than not. In the same manner, all power, of whatever sort, is of itself desirable. A man would not submit to learn to hem a ruffle, of his wife, or his wife's maid; but if a mere wish could attain it, he would rather wish to be able to hem a ruffle.

Thus spaketh Samuel Johnson, as quoted by James Boswell in Life of Johnson.


Ten years ago, I was late to the inline skating party, but once I was there, I threw myself into it. Once I had taught myself how to fall and how to stop, well, there was no stopping me. I trained for marathons, often skating 70 miles or more a week.

What I never did learn, though, was how to maintain my skates. Having grown up in the company of mechanical geniuses, I believed that the wielding of tools was far beyond my ken. Not to mention the learned helplessness that develops when one is in the habit of allowing others doing for one what one might be capable of doing for oneself if one only took the trouble.


Relying on the kindness of another eventually became an insurmountable obstacle. Doesn't that sound stupid? But I quit skating--something I loved--because of my combined ignorance and unwillingness to learn how to maintain the skates myself: if you don't clean the bearings and rotate the wheels, you slow down, which deceleration drastically decreases the fun and increases the risk (as counter-intuitive as it sounds, the slower you go, the less stable you are).

Last weekend, I persuaded someone to show me how to take the skates apart and clean them and put them back together. Then I did it. The process took hours and hours. I had grime under my fingernails (what was left of them) and a streak of black grease on my face and I was tired. But by the time I was done, I thoroughly knew how to do this thing. There is nothing so minute or inconsiderable, that I would not rather know it than not.


So I learned how to do this task that did not come easily to me, one to which I used to have aversion, because I was highly motivated:

There are certain aspects of tasks that induce greater effort and persistence: a performer’s interest in the task and the level of difficulty of the task. These factors are relevant in creating an environment where an individual is likely to exert more effort and, in turn, become more industrious. Therefore, task interest and task difficulty may both act as moderators in the relationship between effort and industriousness.

Which effort was reinforced:

Learned industriousness theory asserts that reinforcing an individual for achieving a performance standard increases the likelihood of that individual performing those behaviors again. If the individual exerted high levels of effort during the completion of the task, the effort takes on its own reinforcing value. This is because the individual enjoys the sensation of working hard because it is associated with reinforcement. Therefore, this individual is more likely to generalize this high level of effort to other tasks because it is less aversive and is associated with positive results. 

The joy is in the doing.


You understand that this is really about how we can figure out how to help kids access that joy of exerting oneself and developing competence and confidence--in whatever realm.


What is the lever that triggers the engagement?

How to Get Good at It (Whatever It Is)

Maybe this is evidence of brainwashing from my college years, but I hold fast to the belief that the road to happiness is to become really, really good at something. How pleasant if that thing is also something at which one can earn a living. Even if not--even if that thing is carving sculptures out of butter or that strange combination of art, vocation, and drudgery of being a parent or if it is a kind of play--training one's dog, growing orchids (not my area--I'm just now trying to resurrect several that I nearly murdered from neglect followed by equally damaging obsessive attention), or building sculptures out of buttons--it's still not only a worthwhile pursuit but the highway to heaven.

Why?

1. Fun--Fun is absorbing, leads to flow, there are brain wave changes, look it up.


2. Becoming a master of something changes you deeply. You develop the confidence of the expert, which has a salutary deflating effect on the ego, thereby creating room for curiosity. You can afford to admit ignorance and to consider that you have something to learn. When I was studying Iyengar yoga and was admitted to the invitation-only level IV-V class, any pride I might have had was extinguished by the obvious truth that I was the dunce of the class. This was incredibly liberating. I felt freed of any expectations I might have had, any desire to compete (oh, people say yoga isn't competitive--baloney! If humans do it in a group, someone is going to try to dominate or show off, probably many someones) simply because I lacked the ability and experience to be able to compete at that level. Some of the people in the class had themselves been teachers for many years, others had studied in India, some had had a yoga practice for decades.


3. The joy is in the doing--not in feeling special for the doing, not in the attention one might get for the doing, but in the doing itself.

But one only learns this joy if one finds something that one really really loves to do, and then works and works and works and works at it. Malcolm Gladwell says it takes 10,000 hours to achieve mastery.

I think about kids at school. How do they get a chance to find what they love to do and are really really good at? Some maybe just stumble across it. One of the protegees (and I desperately hope she doesn't mind I'm talking about her here) is shockingly good at content development. She's a natural. She just has this way of thinking, this combination of acuity, precision, and creativity that makes her a knockout. With only a few months experience, she's creating work of the quality I'd expect from someone with years of experience. Did she choose a career in assessment content development? No. It was sort of happenstance facilitated by a recommendation of someone who is a friend of mine and was one of my protegee's teachers. Never underestimate the power of a teacher to guide, nudge, encourage.

Maybe sometimes kids are really good at things they don't necessarily love, and maybe sometimes they love things they're terrible at. Then what? We can't all be good at everything. We can't all be suited for everything.

When I was in the sixth grade, I'd been reading all the Black Stallion books for years, and more than anything I wanted to be a jockey. I was 5' 6" and weighed 120 pounds and someone hinted to me that jockeys tend to be built more delicately. I was crushed. (I got over it.)


Last night I was making a list of things I'm not just terrible at, but monumentally and breath-takingly terrible at.


I was the world's worst secretary. When I was 21, I burst into tears at a panel job interview when I learned there would be a timed typing test (what's funny is that I'm now quite speedy on the keyboard, writing having become my life's work). I like to cook but once in a while I stop paying attention and something catches fire (generally a sleeve, although I branch out occasionally and have set fire to two wooden cutting boards and a couple of oven mitts--and I'm not even listing all the times I left a kettle on the stove and then went my way). I like to knit but the highest knitting rank I will ever achieve is that of advanced beginner, as I get confused when I see numbers and letters in the same paragraph, and so find knitting patterns impossible to read and also? Directions are boring. I am bad at sitting through anything that bores me. In fact, I am really bad at sitting still. When I'm on the phone, I have to pace or sketch or file papers or make coffee or snip dead blossoms off the rosebush. Once I did sit still long enough knit a sweater. I read the directions and then congratulated myself on my creativity in not following them. We named the sweater Moby Dick. It was of the shape and dimension of a giant hobbit--or maybe an orangutang: too short in the torso, as wide as three rather hefty people, and with arms that reached nearly to my knees. It had a fetching hood of Medieval appearance. I had to wear the sweater, at least a couple of times, because it took probably 100 hours to knit, and when I did, I looked like a deranged monk. However, the sweater was banished to the Goodwill because it endangered my life when I forgot about the billowing sleeve while making coffee. Yes, it caught fire. TWICE. You couldn't tell after I brushed off the charred yarn. I knitted a skirt, too. The skirt never caught fire, but you'd think it would have spontaneously combusted from sheer hideousness. It looked like a homemade tent fashioned from olive, orange, and purple yarn barf. I just threw it away, although it might have been a serviceable sleeping bag if I'd just sewed the hem together. I GET LOST. Getting lost is so deeply embedded in my life that I have a formula for how much getting lost time to allow depending on the distance I'm traveling: For less than an hour's drive, I allow 30 extra minutes; for more than an hour, 60 minutes; for longer trips, it's another half day. I've gotten lost going to the airport, and to the cello teacher's--where we go once a week and have done for the last two years. I've been late for planes and for job interviews (before the formula). I'm always late to doctor appointments (I've gotten lost going there, too, even though it is 15 minutes away and I think I know where the office is, but my doctor kindly tells me she never minds because it gives her a chance to catch up on her file notes.


UPDATE: Fixed typo & formatting. Same as it ever was.

Just Because

The Secret Service was formed not to protect the president, but to protect the economy. During and after the Civil War, the sun shone brightly on counterfeiters, and they made plenty of hay. The bills were so crude then and counterfeiting was easy to do and difficult to detect. The economy tottered.


Hence the Secret Service. (Maybe there should be a new branch to protect the economy from banks.)


How do I know? Reading.


A few days ago, I read one of the best and funniest short stories I've read in ages, "The Revolt of Mother," by a writer of whom I'd never heard: Mary Wilkins Freeman. (Yes, I am an ignoramus. Though I read a lot and have been so doing for many years, I never made much of a study of American literature. Except the exceptions: Flannery O'Connor. Eudora Welty. The modernists. Charles Bukowski and John Fante. Some others.)


Anyone who's ever been married for a long time will understand this: 

“I wish you'd go into the house, mother, an' 'tend to your own affairs,” the old man said then. He ran his words together, and his speech was almost as inarticulate as a growl.

But the woman understood; it was her most native tongue. “I ain't goin' into the house till you tell me what them men are doin' over there in the field,” said she.

I won't tell you what happens.


It was in a volume of great American short stories, which volume included all the usual suspects: Hawthorne, Hemingway, Fitzgerald, great storytellers all.


What else am I reading? Marianne Moore's Complete Poems and I'm rereading Stet: An Editor's Life by Diana Athill.


What I really want to be reading, too, but I loaned it out or misplaced it: Offshore by Penelope Fitzgerald. That book is just a lovely afternoon.