Wednesday, February 29, 2012

Rush to Judgment?

Not such great news from Paul Fain at Inside Higher Ed:
Large numbers of community college students are being placed into remedial courses they don’t need, according to new studies that questions the value of the two primary standardized tests two-year colleges use to place students: the COMPASS and the ACCUPLACER.
I find this news surprising. I taught both remedial and first-year English at two different community colleges (as an adjunct, like Professor X, who was called "an academic hit man" by the NYT, though I don't endorse nor share all of his opinions and "lemony plaintiveness," which you can read about here).

My teaching career was short-lived, but I loved teaching at community colleges. I loved most of my students. The young ones made me laugh. The older ones worked hard. (Some of the young ones worked hard, too--but the older ones were clearly on a dedicated mission to improve their lives, and having worked many years at jobs beneath their abilities, had identified education as the highway to heaven a better life.) They all of them sometimes surprised me with their storytelling--in a good way, once they figured out they could be themselves in their writing. In fact, I often wish I could have the opportunity and the financial independence (teaching jobs in this coastal area are as highly sought after as they are poorly paid, it's such a privilege to live near the beach) to allow me to return to the classroom.

However, the students in my remedial classes definitely belonged there. Their need for remediation was so great that at first, I wasn't quite sure what to do with them. Truly, I had seen better writing from fourth-graders--and I don't mean fourth-grade prodigies, I just mean regular little fourth-graders. My students in the remedial classes could not write a coherent sentence--even a simple declarative one--and so that's where we started. By the end of the semester, many would be writing decent paragraphs. Not all. Not even most. I don't know if they were ready to go on to college-level classes in every respect, but even the ones who still needed improvement were more ready than they had been. But a lot of them had dropped out by then, too. That's the unfortunate nature of the beast.

Maybe 25% of the first-year English students were second-language learners. These were the ones that troubled me most. (Except that one really smart girl who had to bring her baby to class because her childcare arrangements kept falling apart. I worried about her a lot, too.) They didn't understand me when I talked, so during class, they would pull out calculators and work on assignments from other classes until the weight of peer pressure (I would stop talking and look at them and the other students would fidget and grumble) bowed them into submission, and they would sit and look at the floor praying for release from what must have been unbearable tedium.

I don't know how the second-language learners were able to pass the placement tests, but pass they did. Their research papers were often bought from those sleazy Internet sites. It didn't take much detective work to figure that out; I always had my students write a lot in class.

This was a long time ago, though, at the very beginning of the commencement of my work in assessment (so I had no idea about and even less interest in the placement tests) and all is shrouded in mystery. Anyone who's ever taught the basics at a community college will probably agree with the necessity of some kind of placement tests. Grades are an insufficient and often wildly inaccurate measure; some of my remedial students reported achieving As and Bs all through high school, which achievement must have been the result of their brute charm, rather than any mastery of academic content.

According to the author of the study, "students who ignored a remedial placement and instead enrolled directly in a college- level class had slightly lower success rates than those who placed directly into college- level, but substantially higher success rates than those who complied with their remedial placement, because relatively few students who entered remediation ever even attempted the college-level course."

I'm going to spend more time with this study. Maybe there is the potential for flawed logic in the leap to "this raises questions not only about the effectiveness of remedial instruction, but also about the entire process by which students are assigned to remediation."

Sure, I'm open to that possibility. But there are other possibilities that may be overlooked in the rush to indict. What if the students who ignore the remediation recommendation are the self-selected most highly motivated do-or-die students? What if the students who follow the remediation recommendation are the ones who don't have the time, interest, inclination, motivation to slog through years of night classes while working full-time?

Just something to think about. I have no dog in this fight. If the tests do not do what they are supposed to do, by all means, let's use a measure that is accurate and fair to students.

Tuesday, February 28, 2012

File Under: Charity Begins at Home, Cross-Referenced to Double-Edged Sword of Celebrity and the Madonna Channel

Here it is, the curse of celebrity. Everybody knows everything you do, so danged if you dodanged if you don't.

Even these days, at a time in Her life when Madonna has set aside the shenanigans hijinks nonsense antics of Her colorful youth, She continues to make headlines.

For the record, I was an unabashed Madonna fan until I wasn't.

Whether or not one admits to being a fan, one must admit that She has the great gift of shameless self-promotion reinventing Herself. As She herself freely stated in the sort-of documentary Truth or Dare, She's not the best singer nor the best dancer in the world, but She is mightily blessed with blonde ambition.

I find the news of Her plans to build 10 schools in Malawi rather curious, in spite of my belief that what is good must be good. How can it be a bad thing to build schools? And yet, one can surely sympathize with the chagrin expressed herein:

But I wonder whether She had opportunity to consider that there are plenty of schools in Her native country--nay, even in Her native state--that could use a little help.

On the other hand, how many of us have the means (and the hubris) to march into another country and anoint appoint ourselves the new patron saints of education with nary a by your leave, sir to the national secretary for education, science and technology?  (Although upon reflection and having met myself before, I am 100% certain that if I were possessed of the former, I would absolutely make a fool of myself in a million billion gazillion heretofore unimagined ways with the latter.)

Monday, February 27, 2012

Ringside

This debate about the Common Core Standards is one in which I have a deep and abiding interest. In the last year, much of my content development work has been to write (or edit or review) materials addressing these standards.

As standards go, what I can say is that they're fine. Nothing unusual. Nothing at all outside of the realm of what is commonly assessed (here I speak only of ELA as I'm not qualified to evaluate the soundness of the math standards from the perspective of a content area expert, but my math colleagues have told me their opinion of the math standards is similar). We all agree that the standards do bear the smudges from many sets of fingerprints--and that this is to be expected from anything produced by a committee, our experience being that members of committees often come to the table wheeling bearing their own baggage agendas, and much compromise is needed in order to reach the Promised Land consensus. 

However, we none of us think that there is a monster hiding under the bed anything to suggest a federal brainwashing campaign conspiracy hijacking of state and local authority in the schools.


Arne Duncan employed a Jab, Right Hand, Left Hook in his response (which I paraphrase below): 

Being just a lone voice crying out in the wilderness an independent consultant unaffiliated with any government office or department (or any publisher), of course I am not privacy to the politicking. There is probably more here than meets the eye.

On a completely separate note, in re high school exit exams and college readiness, I hear from the Center on Education Policy (CEP):
27 of the 31 states with high school exit exams reported participating with Partnership for for Assessment of Readiness for College and Careers (PARCC) and/or SMARTER Balanced Assessment Consortium (SBAC) to develop common assessments that are aligned to the CCSS [Common Core Standards] to measure college and career readiness. Of these 27, at least 16 states plan to replace their current exams with consortia assessments. . . .
[In the spirit of transparency rather than bragging, I tell you that an article I wrote for National Geographic Explorer, "Seeing Eye to Eye" was reprinted on page 74 of The Common Core Standards for English Language Arts & Literacy in History/Social Studies, Science, and Technical Subjects Appendix B: Text Exemplars and Sample Performance Tasks. The reprint permission request went through National Geographic, so this was a fait accompli long before I received the unexpected reprint check, and much (if not most) of the credit should go to my editor at NGE for reasons previously discussed.]

Friday, February 24, 2012

Tell Me Something Good

. . . in the immortal words of Chaka Khan.
More than 2.5 million students in New York City, Charleston, Atlanta, Indianapolis, Jersey City, and Los Angeles have benefited from the Overcoming Obstacles Life Skills Program. The program is designed to help educators teach communication, decision making and goal setting skills, aimed at giving middle and high school students the skills to be successful in life.
More here. By B.A. Birch at Education News. Overcoming Obstacles offers free lesson plans for middle school and high school.

Happy Friday.

Thursday, February 23, 2012

Devices of Wonder

--it's a fun interactive online exhibition from the Getty.

There is a lot of fun out there on that Innerweb Internet. Isn't it great that there is value in play?


More art for the classroom:

Wednesday, February 22, 2012

The Beautiful and the Confused

California is like the shockingly gorgeous woman at a party who desperately wants people to think she's not just a pretty face with cleavage a lovely landscape and so is always coming out with the polysyllabic words (the production of which seems almost painful) in the hopes that someone will finally take her seriously and stop staring at her cleavage Sierra foothills.

Someone needs to tell her that she really doesn't need to try so hard. Although maybe it is time she figure out the difference between correlation and causation.

In the late 1980s and early 1990s, it was the self-esteem campaign with a task force and everything. (You can read the full report here.) Even though all the other states like to make fun of California and New York (she being so pretty and he being so cool and rebellious), as they go, so goeth the nation. The promotion of self-esteem in children snowballed into almost a religion. Praise became as constant and unrelenting as it was--I'm sure--meaningless to the intended beneficiaries.

As with any method of symptom-mowing, that didn't quite work out as anticipated:
The long-term impact of this rah-rah mentality is already apparent. In 2004, according to Jean Twenge, author of Generation Me, 70 percent of American college freshmen reported their academic ability as “above average.” But, once ego-inflated students get to college, they’re more likely to drop out, says Twenge, when their skewed sense of self and overconfidence affects their ability to make decisions.
Because they got it backwards. Although self-esteem and high student achievement may be correlated, the cause-and-effect sequence is more likely to be that high student achievement promotes self-esteem, rather than the reverse. *Oops. Dang. How much did we spend on that report again?*


(Oddly enough, in a six-degrees-of-Kevin-Bacon six-degrees-of-separation kind of coincidence, just when this self-esteem campaign was a little sparkly idea glimmering on the horizon, I was working as a secretary at an automobile repair shop that was owned by John Vasconcellos's brother. There simmered family tensions, is all I have to say about that. Also, such work did not enhance my self-esteem, as the owner's wife was in the habit of remarking that how interesting that I was a college graduate--she had never attended college--and yet I didn't know how to load paper into a printer. I didn't blame her; no amount of praise could have convinced me I was even a minimally adequate secretary. I was a terrible secretary, maybe the world's worst secretary, with deplorable office skills that were on a par with my knowledge of auto repair. One of my duties was to translate the mechanics' notes on the service orders into descriptions of labor  for the customers' bills. Once, after replacing a part, a mechanic scribbled "lower radiator hose" on the service order. I wrote "lowered radiator hose" on the final bill, which ignorance sort of rendered everyone, from the high school kid who pushed a broom around to the owner, speechless. )

I would like to point out that it's antithetical to serious intellectual inquiry to hijack such a discussion by misrepresenting the "basic premise" in order to promote a political and social agenda:
The basic premise is that racism and discrimination cause minorities to feel bad about themselves, and that this low self-image translates into women avoiding "hard" fields like engineering and blacks and Hispanics doing poorly in school.
Well, no, not exactly. The basic premise is that people--and let's leave race and gender out of it, shall we, because neither has much to do with the main point--who feel bad about themselves for whatever reason, tend to self-destruct--and take others with them--which can only have negative effects on not just society, but the economy, so how might it be possible to kill the snake when it is young address this problem in children so that they are able to become self-sufficient adults who are well-equipped to function and thrive, which will mean less crime, fewer teen-age pregnancies, less substance abuse, and more productivity, which will mean more money for big companies and the local and federal governments that tax them. The means may have been ill-advised, but the goal seems like one we could all get on board with to some degree of enthusiasm, if indeed it be based in fact.

Similarly, the motivation driving the California Department of Education (and those of Michigan and Oklahoma and others) to dissect creativity in order to figure out how it works and thus build it into school curriculum has to do with money business productivity. This isn't a new idea:
The world community recognizes that progress in the arts, in the professions, and in science and technology relies exquisitely on the creativity of people in these professions.
This came from Carl A. Leopold (Boyce Thompson Institute, Cornell University, Ithaca, New York), who went on to suggest
. . . the art of scientific thinking be taught by allowing students to experience all the thrills--and missteps--of an actual science program or research.
Which indicates that independent exploration and direct experience and willingness to fail are essential to creativity. It's an orientation from which to teach, rather than a framework for a curriculum.

Once creativity is allowed in, students may gain mastery, thus building self-esteem without anyone heaping on piles of praise--praise that the students are surely smart enough to recognize as false. Everything is everything.

Tuesday, February 21, 2012

You Got to Tip on the Tightrope

. . . in the immortal words of Janelle Monae.





In re California legislation S.B. 789

This bill would require the advisory committee to consult, as appropriate, with individuals who are experts or have experience in innovation in the fields of business, science, technology, mathematics, engineering, and arts education on the development of a voluntary Creative and Innovative Education Index, to be based in part on the creative opportunities in each participating school, as specified. The bill would require the advisory committee to make recommendations by June 1, 2013, to the Superintendent on the extent to which this index should be part of the state’s accountability system and methods to foster creative and innovative education in the public schools.


Does it make you nervous  to think about legislators regulating creativity? It seems as disagreeable as the prospect of the business folks messing with the talent.


But maybe "regulating" is too strong a word; the bill started out like this, was amended to this, and then to this. Perhaps it's more accurate to say "considering implementing recommendations about"?


My first thought is that when deciding whether to follow a recommendation, one must carefully consider the source.


Dr. Mihaly Csikszentmihalyi suggests that there is a creative personality.


Can creativity be taught? Or should we think about creativity as the seed of a flower that must be nurtured in order to bloom? If so, what conditions will encourage that little flower?


When in doubt, I turn to Einstein:
The really valuable thing in the pageant of human life seems to me not the political state, but the creative, sentient individual, the personality; it alone creates the noble and the sublime, while the herd as such remains dull in thought and dull in feeling.
Italics mine.


Meanwhile, if (in addition to allowing them some autonomy, some freedom to experiment with what they think) we can give kids music to listen and dance to, cool stuff to mess with, and beautiful things to look at, they'll figure out what to make of it all.


They need as many opportunities to wonder as possible:
The most beautiful experience we can have is the mysterious. It is the fundamental emotion that stands at the cradle of true art and true science. Whoever does not know it and can no longer wonder, no longer marvel, is as good as dead. . . .


Are you wondering about the tightrope? Oh, it's balance, and the need to take risks. Edwin Land said, "An essential aspect of creativity is not being afraid to fail." We all of us need to be able to boldly go without worrying about what will happen when--not if--we fumble. Fumbling isn't failure; it's part of learning.


UPDATE: Wish I'd seen this earlier. From Namaste Nancy, the reminder that creativity is for everyone.

Monday, February 20, 2012

File Under: All Roads Lead to Rome, Cross-Referenced to Mandatory Reading

The book every K-12 content developer--assessment and curriculum--should read is Tested: One American School Struggles to Make the Grade, which was recommended to me by a colleague and likeminded comrade in quality assessment content development, Carmen, a senior level genius expert at Anonymous Testing Company.


To say that the tests administered yearly at grades 3-8 (inclusive) are high stakes cannot possibly begin to convey what this means for students, teachers, and school administrators and how NCLB has transformed the educational landscape. The flames are licking at their feet every minute of every day.


And even when we who are informed are talking about this, and about how awful it is that teachers must teach to the tests and that the assessments are driving the curricula--it's one thing to talk about it, and it's another thing to experience it.  If you're not a student, teacher, principal, or parent, this book is the closest you can get to the fire.


Like the children in Tested, my daughters' skills and knowledge are assessed so frequently at school (and what is taught is often so narrowly focused--the algebra teacher actually labels each homework assignment with the assessable standard and tells parents that she does this so she will know whether students will answer those questions correctly on the state test) that I'm shocked by how little genuine instruction they actually receive and so I supplement their classroom instruction by offering my own reading, writing, social studies, and science lectures. Which may bore them nigh unto death, who knows, but I refuse to send them out into the world as little ignorami ignoramuses. (For math homework help, we turn to my friend and cohort and math content area genius expert Carrie Frech, who works at a major testing organization).


My daughters came home last week emitting little puffs of indignation over the latest district benchmark assessment. (They swiped it and brought it home to show me.) When I read it, I was horrified. It was a passage-dependent writing prompt. I didn't see a rubric, God knows what horrors hide behind that curtain, but I assume the responses were scored for reading and writing.


The story, a tedious adaptation of a folktale, was poorly written and at the fourth grade level (this, for an eighth grade gifted and talented program; my daughters are currently reading The Great Gatsby and Rebecca for their next book reports, and yet they're being assessed with text suitable for fourth grade?). What was there was was presented at an extremely literal level of understanding. Nothing in the story allowed for any genuine analysis of narrative elements nor interpretation of literary devices, and yet the writing prompt required the students to do just that. I don't know how they could. You can't make a pie out of one apple. The multiple-choice section (developed by a company relatively new to the game for whom I'd done some work a few years ago and by whose lack of understanding of test development at that time had shocked me) was no better. The girls told me that there was one question that was so nonsensical that, districtwide, the teachers' form of protest was simply to give all of their students the answer. Does anyone see any value whatsoever in the use of such an assessment tool?


When I did some work for this company a few years ago, I observed their inexperience with and lack of knowledge about assessment. Maybe things have changed since then, I don't know. What I do know is that this company--the same one that has little assessment background--doesn't perform any field testing of the test content and there is no data whatsoever to indicate that we can make any kind of accurate inferences about what students know and can do based on such poorly constructed assessments. But this company does a good job of selling, and the districts buy the dream fantasy idea that the products will 1) give teachers information about their students that will help them get students ready for the state test and 2) predict how students will perform on the state test. Neither claim is possible, particularly with assessments that violate the most basic quality standards.


All roads lead to Rome; it all comes back to the Quality Manifesto.


(And it must be said that certainly there is room for quality improvements in the classroom as well, and there was room even before NCLB. What passes for instruction in some classrooms horrifies me just as much as any quality train wreck I see in the assessment world. Two teachers at my daughters' school ROUTINELY play audio of the textbooks instead of teaching--this, in science, which I think we can all agree requires hands-on instruction; it's still bad in reading, but not quite so bad). One of these teachers also ROUTINELY spends twenty minutes or so of the fifty minute class period discussing her personal life to the captive audience of eighth-graders--they know all about her kids, her husband, her political views, her hobbies, her extended family, and her domestic habits. So do I. Unless someone is a friend, loved one, or crazy person celebrity, there's probably not much in her life you want to hear about for twenty minutes straight, and yet that is what this teacher subjects her students to instead of teaching them.)


UPDATE: Identified my book-readin' comrade by name. Thanks, Carmen!
UPDATE: Added a link.
UPDATE THE THIRD: Removed a link, anonymized an identity.

Friday, February 17, 2012

Why Not Offer Miracles If You Can?

Miracles
WHY! who makes much of a miracle?
As to me, I know of nothing else but miracles,
Whether I walk the streets of Manhattan,
Or dart my sight over the roofs of houses toward the sky,
Or wade with naked feet along the beach, just in the edge of the water,         5
Or stand under trees in the woods,
Or talk by day with any one I love—or sleep in the bed at night with any one I love,
Or sit at table at dinner with my mother,
Or look at strangers opposite me riding in the car,
Or watch honey-bees busy around the hive, of a summer forenoon,  10
Or animals feeding in the fields,
Or birds—or the wonderfulness of insects in the air,
Or the wonderfulness of the sun-down—or of stars shining so quiet and bright,
Or the exquisite, delicate, thin curve of the new moon in spring;
Or whether I go among those I like best, and that like me best—mechanics, boatmen, farmers,  15
Or among the savans—or to the soiree—or to the opera,
Or stand a long while looking at the movements of machinery,
Or behold children at their sports,
Or the admirable sight of the perfect old man, or the perfect old woman,
Or the sick in hospitals, or the dead carried to burial,  20
Or my own eyes and figure in the glass;
These, with the rest, one and all, are to me miracles,
The whole referring—yet each distinct, and in its place.
  
To me, every hour of the light and dark is a miracle,
Every cubic inch of space is a miracle,  25
Every square yard of the surface of the earth is spread with the same,
Every foot of the interior swarms with the same;
Every spear of grass—the frames, limbs, organs, of men and women, and all that concerns them,
All these to me are unspeakably perfect miracles.
  
To me the sea is a continual miracle;  30
The fishes that swim—the rocks—the motion of the waves—the ships, with men in them,
What stranger miracles are there?
                                                  Walt Whitman
"To me, every hour of the light and dark is a miracle." To me, this poem is a miracle.


I could spend hours looking out the window and thinking about this. I did spend hours thinking about the poem at night as I walked my dog in the canyon and looked up at the stars and heard the breeze brush through the palms and in the morning as I took my daughters to school and we crested the hill and caught sight of the ocean and the islands and in the evening as I took out the trash and saw the sunset so garish that if it were a painting, you'd make fun of the artist.


I came across this poem because recently I had the great good fortune of having the opportunity to think about how to bring the American Romantics into certain high school classrooms in a certain state (intentional vagueness required by both professional courtesy and the stipulations of the non-disclosure agreement that is part of my contract).


For a week, I read Emerson, Lake, and Palmer Emerson ("To believe your own thought, to believe that what is true for you in your private heart is true for all men, — that is genius"), Whitman, Dickinson, Dunbar. Leaving out Thoreau wasn't intentional; it's just that I started with Emerson, but once I was in the thick of the poetry, decided to stay there and tackle the essays another time.


This work was pure pleasure. Poetry and grade 11 are a match made in heaven. All those emotions, for one thing. And there's a poet for everyone. Baudelaire was the original Emo:

Dark one, I am torn
By your savage ways,
Then, soft as the moon, your gaze
Sees my tortured heart reborn.
                                      --from "Afternoon Song"


Not the least of the pleasure was my thinking about the students. Maybe there would be one or two who for whom these poems might be a signpost to the onramp to the highway that leads to this gorgeous world of poetry and self-knowledge.


Last week, I went to hear a friend and fellow alumnus from the College of Creative Studies at UCSB  speak about literature and read from his writing. He told the students about how, when he graduated, he had this sense of having a gift, a talent, that was bigger than he knew what to do with.


It was true of all of us, I think. I'll go further and say that it may be true of everyone, but maybe everyone is not lucky enough to understand that he has a gift, or not lucky enough to land in a place where he has the room and space and encouragement to find and exercise his gift.


My daughters sweetly and patiently listen when I talk about my work and endlessly quote from my readings. They have more than a passing familiarity with Emerson by now. Erin, my eldest, showed me what her social studies textbook says about Emerson and Thoreau.


The targeted standard is to read the writings of the American Transcendentalists. The means employed by the textbook writer to address this standard was to write one spare paragraph about these two in which Thoreau is described as someone who was jailed because he refused to pay a one-dollar federal tax and Emerson is described as someone who didn't want to go to jail and so he paid the tax. That's it. Thus summing up the philosophies and values of American Transcendentalism. There was no context, no real biographical information, there were no excerpts from the essays.



Why in the name of all that is holy would a writer (or a publisher, as surely it was not solely the writer's decision--as a writer, I often bump into the decisions of editors and publishers) pass up the opportunity to offer more?

In the immortal words of Emily Dickinson (see below), the brain is wider than the sky. Even at 14. I'm sure their brains could not only accommodate bushels, nay, truckloads of real information, rather than a dismissive tag line. Why not offer the miracles and let the students sort them out?



CXXVI
THE BRAIN is wider than the sky,
For, put them side by side,
The one the other will include
With ease, and you beside.

The brain is deeper than the sea,
For, hold them, blue to blue,
The one the other will absorb,
As sponges, buckets do.

The brain is just the weight of God,
For, lift them, pound for pound,
And they will differ, if they do,
As syllable from sound.

UPDATE: Fixed link to College of Creative Studies at UCSB.

Tuesday, February 14, 2012

Big Wheel Keeps on Turning

This just in, from Frank Brockmann, Center Point empresario and fellow quality crusader:

President Barack Obama announced last week that 10 states will be exempt from the requirements of the highly-criticized No Child Left Behind legislation. In exchange, those states will have to agree to a series of reforms. But some experts say the law should be scrapped completely for models that don't rely on standardized tests.

Interesting, because last night I was reading about the rise and fall of the Roman Empire California Learning Assessment System. CLAS was not the first--and surely will not be the last--testing program to be felled by public controversy. Among the many objections to CLAS were to the reading passages (two of Alice Walker's stories were excised after complaints that "Am I Blue" promoted vegetarianism, and "Roselily"was anti-religion), to the writing prompts (which were considered invasive), and to the test questions themselves (too subjective and based in emotion, according to critics).

When I began my career in hand-scoring, CLAS was one of my first projects, so I did at one time have more than a passing familiarity with the tests and how students responded to them. In general, I liked the questions (what did I know? I hadn't any content development experience at that point), although scoring did present a problem.

In some parts of the test, students were encouraged to take notes--which were called "marginalia"-- and for some questions, were offered various options as to how to respond: they could draw a picture, for example. Anyone who's ever been presented with a drawing by a small child understands the obstacle in scoring there:

You: Oh, what a lovely dinosaur.
Child: It's not a dinosaur.
You: No?
Child: No.
[pause]
You: What is it, love?
Child: It's a manatee.
You: Oh, yes, of course it is.

I see what the developers and supporters of CLAS were attempting to accomplish, and the goal is a laudable one: to lure students into engaging with authentic literature and then to welcome their genuine, individual responses to what they read. However, in retrospect, these goals might be more readily achieved at the classroom level, through both instruction and formative assessment (and by "formative assessment," I really mean all those teacherly techniques for paying attention to one's students, not the administration of a series of badly-written multiple-choice quizzes), than through a standardized testing program.

The point is really to define clearly one's purpose, and then create the assessment that will best serve that purpose, as "the man without a purpose is like a ship without a rudder," in the immortal words of Thomas Carlyle.

Big wheel keeps on turning. We went from bubble-in to performance assessment, back to bubble-in, and now we're talking about performance assessment as if it were the hot new thing, never before attempted.

Not that we shouldn't move in that direction--I like that direction very much--but if we do, there should be a moment of planning and taking stock (For which of you, intending to build a tower, sitteth not down first and counteth the cost, whether he have sufficient to finish it?), a gathering of the elders for the harvest of wise counsel, before there is a marshaling of forces. What is our purpose? What will we gain, what will we lose, is the loss worth the gain? How do we prepare teachers for this role?

What do you think?

Monday, February 13, 2012

In Defense of Quality

If you cannot learn to love real art, at least learn to hate sham art.

This, from William Morris.

By “real art,” let’s say James Lesesne Wells, for example, and by “sham art,” let’s say Thomas Kinkade

I’d like to apply this sentiment to the work of writing, and more specifically, to the work of test content development. Quite frankly, I’m mystified (a phrase I borrow from an assistant district attorney with whom I used to work when I was on a very different career path, and who was in the habit of using this phrase to sharpen his tongue as he prepared to slice me up for having done something with which he disagreed) by not only the deep and devastatingly obvious diminution of quality in test content in the past few years, but also by the failure of people in this silo of the industry to recognize this trend.

Quite frankly, it breaks my heart.  As silly as it sounds. But when you love, you expose yourself to the risk of heartbreak. Again, I turn to William Morris, who said, “Give me love and work – these two only.” I’ve been doing this work for 19 years now; though I got into it thinking it was a temporary rope to keep me out of the quicksand until I found my magic circle niche place on solid ground, I think we can all agree it’s become a long-term relationship.

If this lack of quality trend were limited to newcomers to the business, we could propose that them entry-level young’uns [*sigh*] are poorly educated and ill equipped to express themselves except via texting, which you can certainly see in their editorial comments (which are lamentably rich in acronyms, emoticons, and which betray an unfortunate juvenile fondness for excess punctuation and using all caps in directions, which cannot help but set one’s back up, however patient one might be, and anyone who knows me knows that overly patient I be not).

But no, all we content dev folk – ELA, math, science, and social studies, not one is immune, no, not one – have noticed, and we do talk about it, and the conversation and all the various repetitions and iterations of the same conversation bore and horrify us so that we are reduced to shaking our heads and turning our attention to some vision of an oasis, such as the cocktail that awaits the end of the day.

Back in the day, when I worked at Great Big Huge Test Publishing Company, I went to a mandatory training on root cause analysis. We used the fishbone chart. As trainings go, it was all right. Certainly better than the one at which I was accused of not doing my work and letting my teammates pick up my slack because I failed to participate in the assembling of a puzzle, which failure actually had a lot to do with my abysmally poor spatial intelligence and equally poor vision (since corrected through the wonders of Lasik surgery) and little to do with my work ethic, which, as it happens, is about as Puritan as a work ethic might be. You can take the girl out of the working class, but you can’t take the working class out of the girl. But I digress.

If we performed a root cause analysis on the wreckage of Good Ship Quality, what would we find?

To answer that we’d have to go back to the beginning. When I started as a content editor, I was dedicated to one project. That project was my one, my only, my all in all. It was the same for my co-workers. That was the early 90s. In most states, large-scale tests were restricted to reading, writing, and math, and were administered at three or four grades (usually something like 4, 6, 8, and 10, or 5, 7, and 11).

Five years later, it was a whole different and bigger but not necessarily better ballgame. More states were testing more grades, and NCLB loomed on the horizon. As a supervisor, I was responsible for five projects. No one on my team was solely dedicated to any one project; each person, from editor to supervisor, worked on several.

I had a meeting with my manager that went like this:

Manager: [peering at her clipboard] All right, so you have State V, State W, State X, and State Y.
Me: And State Z.
Manager: Oh, I forgot about Z. Right. State Z. [scribbles a note on her clipboard]
Me: What is the order of priority?
Manager: [pause] They’re all priorities.
Me: With five states, mistakes are going to be made. It’s impossible to supervise five projects of this scale. Which state is going to be the mistake state?


Test publishing companies couldn’t handle the workload. Companies that had never done any testing smelled the money and jumped into the fray. All companies got hiring fever. By then, I was a content development manager hiring entry-level candidates at more than twice my starting salary as an editor (and did that ever sting, I tell you what).

But the equation for meeting a production deadline is

TIME + WORKERS = MEETING PRODUCTION DEADLINE

If you have less time, you need more workers. Fewer workers, you need more time. I am no math expert, but this equation I know.

Deadlines got more and more compressed, development cycles shrank, and everyone starting skipping steps. Real training gave way to on-the-job training, which really means sink-or-swim training. New-hires were handed the comprehensive binder containing lists of processes and procedures, which binders were relegated to shelves in cubicles because no one had time to read them. Early field tests were cast aside. Sometimes all field tests. There were fewer internal reviews. The few remaining reviews were performed by overworked and/or underexperienced staff—and you can actually determine which is which (and which is both) when you see the editorial feedback coming out of such reviews.[1]

Another significant factor may be a corollary to the Peter Principle. The most highly skilled, knowledgeable, and experienced line staff keep getting promoted to management, where they may be doing a fantastic job, but their spots are filled either by new hires or old hands who are left behind (how can I say this delicately? Their remaining behind may not always be by their own choice). Combine this with the absence of training, and it’s a chaos cocktail.

Not to mention the dependence on freelancers. Companies started laying people off and then rehiring them as subcontractors. For some, it’s a win-win—the company don’t have to pay your benefits, and you get to work at home in your pajamas—but it do mean there are a heck of a lot of people at their keyboards writing test questions who neither have experience in education nor in publishing, let alone assessment, which some of us choose to believe is both an art and a science.

There is value in enduring years of slogging through the entire publishing cycle from first draft through bluelines over and over and over again. There is value in having logged many hours in the company of small children struggling to read. There is value in meeting with what the industry calls the stakeholders—teachers, administrators, community leaders, DOE officials. There is value in stretching to accommodate the demands of the stakeholders. There is value in educating oneself about the history and practices of one’s profession. Those learn-to-play-the-piano-in-10-minutes books aside, there is no shortcut to attaining mastery in anything.

There are so many facets to what we do in assessment content development, and when one’s experience is restricted to one tiny mirrored triangle of the great big disco ball, well, that creates a problem because one hasn’t constructed a greater context which allows for greater meaning to inform and guide the work. When the work is simply writing questions for a paycheck and meaning goes out the window, the questions get lamer and lamer, by which I mean trivial, superficial, and plagued by error.

However, the purpose of identifying a problem is not to castigate wrong-doers, nor to enjoy that most basic human pleasure of being right, but to use such identification to find a solution.

The answers are probably as clear to you as they are to me:

  • 1.     Only hire content developers (freelance or in-house, I have no axe to grind here) who either have a proven track record of providing high-quality work or who have the capability (combination of education, writing skills, content area expertise, intelligence, creativity, and persnicketiness) to learn how to do the work well
  • 2.     Provide not only adequate but excellent training
  • 3.     Employ senior content development personnel [*ahem* not naming any names] to review items and provide specific instructional feedback to writers
  • 4.     Budget sufficient time and money for the given project



[1] Overworked but highly experienced people skim text, which forces their brain to fill in the gaps. Which means erroneous assumptions and conclusions drawn from limited evidence and resulting in unnecessary, ill-advised edits. Subtleties or fine distinctions are impossible to detect when skimming. Underexperienced people often restrict their scope of what’s acceptable to their own narrow band of direct experience, and then reject what lay in the outer darkness of their ignorance. This is bad enough on its own, but they will then assume a pedantic tone and lecture the writer for having written items that exceed the demands of the specifications.