File Under: All Roads Lead to Rome, Cross-Referenced to Mandatory Reading

The book every K-12 content developer--assessment and curriculum--should read is Tested: One American School Struggles to Make the Grade, which was recommended to me by a colleague and likeminded comrade in quality assessment content development, Carmen, a senior level genius expert at Anonymous Testing Company.


To say that the tests administered yearly at grades 3-8 (inclusive) are high stakes cannot possibly begin to convey what this means for students, teachers, and school administrators and how NCLB has transformed the educational landscape. The flames are licking at their feet every minute of every day.


And even when we who are informed are talking about this, and about how awful it is that teachers must teach to the tests and that the assessments are driving the curricula--it's one thing to talk about it, and it's another thing to experience it.  If you're not a student, teacher, principal, or parent, this book is the closest you can get to the fire.


Like the children in Tested, my daughters' skills and knowledge are assessed so frequently at school (and what is taught is often so narrowly focused--the algebra teacher actually labels each homework assignment with the assessable standard and tells parents that she does this so she will know whether students will answer those questions correctly on the state test) that I'm shocked by how little genuine instruction they actually receive and so I supplement their classroom instruction by offering my own reading, writing, social studies, and science lectures. Which may bore them nigh unto death, who knows, but I refuse to send them out into the world as little ignorami ignoramuses. (For math homework help, we turn to my friend and cohort and math content area genius expert Carrie Frech, who works at a major testing organization).


My daughters came home last week emitting little puffs of indignation over the latest district benchmark assessment. (They swiped it and brought it home to show me.) When I read it, I was horrified. It was a passage-dependent writing prompt. I didn't see a rubric, God knows what horrors hide behind that curtain, but I assume the responses were scored for reading and writing.


The story, a tedious adaptation of a folktale, was poorly written and at the fourth grade level (this, for an eighth grade gifted and talented program; my daughters are currently reading The Great Gatsby and Rebecca for their next book reports, and yet they're being assessed with text suitable for fourth grade?). What was there was was presented at an extremely literal level of understanding. Nothing in the story allowed for any genuine analysis of narrative elements nor interpretation of literary devices, and yet the writing prompt required the students to do just that. I don't know how they could. You can't make a pie out of one apple. The multiple-choice section (developed by a company relatively new to the game for whom I'd done some work a few years ago and by whose lack of understanding of test development at that time had shocked me) was no better. The girls told me that there was one question that was so nonsensical that, districtwide, the teachers' form of protest was simply to give all of their students the answer. Does anyone see any value whatsoever in the use of such an assessment tool?


When I did some work for this company a few years ago, I observed their inexperience with and lack of knowledge about assessment. Maybe things have changed since then, I don't know. What I do know is that this company--the same one that has little assessment background--doesn't perform any field testing of the test content and there is no data whatsoever to indicate that we can make any kind of accurate inferences about what students know and can do based on such poorly constructed assessments. But this company does a good job of selling, and the districts buy the dream fantasy idea that the products will 1) give teachers information about their students that will help them get students ready for the state test and 2) predict how students will perform on the state test. Neither claim is possible, particularly with assessments that violate the most basic quality standards.


All roads lead to Rome; it all comes back to the Quality Manifesto.


(And it must be said that certainly there is room for quality improvements in the classroom as well, and there was room even before NCLB. What passes for instruction in some classrooms horrifies me just as much as any quality train wreck I see in the assessment world. Two teachers at my daughters' school ROUTINELY play audio of the textbooks instead of teaching--this, in science, which I think we can all agree requires hands-on instruction; it's still bad in reading, but not quite so bad). One of these teachers also ROUTINELY spends twenty minutes or so of the fifty minute class period discussing her personal life to the captive audience of eighth-graders--they know all about her kids, her husband, her political views, her hobbies, her extended family, and her domestic habits. So do I. Unless someone is a friend, loved one, or crazy person celebrity, there's probably not much in her life you want to hear about for twenty minutes straight, and yet that is what this teacher subjects her students to instead of teaching them.)


UPDATE: Identified my book-readin' comrade by name. Thanks, Carmen!
UPDATE: Added a link.
UPDATE THE THIRD: Removed a link, anonymized an identity.

Why Not Offer Miracles If You Can?

Miracles

WHY! who makes much of a miracle?
As to me, I know of nothing else but miracles,
Whether I walk the streets of Manhattan,
Or dart my sight over the roofs of houses toward the sky,
Or wade with naked feet along the beach, just in the edge of the water,	5
Or stand under trees in the woods,
Or talk by day with any one I love—or sleep in the bed at night with any one I love,
Or sit at table at dinner with my mother,
Or look at strangers opposite me riding in the car,
Or watch honey-bees busy around the hive, of a summer forenoon,	10
Or animals feeding in the fields,
Or birds—or the wonderfulness of insects in the air,
Or the wonderfulness of the sun-down—or of stars shining so quiet and bright,
Or the exquisite, delicate, thin curve of the new moon in spring;
Or whether I go among those I like best, and that like me best—mechanics, boatmen, farmers,	15
Or among the savans—or to the soiree—or to the opera,
Or stand a long while looking at the movements of machinery,
Or behold children at their sports,
Or the admirable sight of the perfect old man, or the perfect old woman,
Or the sick in hospitals, or the dead carried to burial,	20
Or my own eyes and figure in the glass;
These, with the rest, one and all, are to me miracles,
The whole referring—yet each distinct, and in its place.
To me, every hour of the light and dark is a miracle,
Every cubic inch of space is a miracle,	25
Every square yard of the surface of the earth is spread with the same,
Every foot of the interior swarms with the same;
Every spear of grass—the frames, limbs, organs, of men and women, and all that concerns them,
All these to me are unspeakably perfect miracles.
To me the sea is a continual miracle;	30
The fishes that swim—the rocks—the motion of the waves—the ships, with men in them,
What stranger miracles are there?                                                   Walt Whitman

"To me, every hour of the light and dark is a miracle." To me, this poem is a miracle.


I could spend hours looking out the window and thinking about this. I did spend hours thinking about the poem at night as I walked my dog in the canyon and looked up at the stars and heard the breeze brush through the palms and in the morning as I took my daughters to school and we crested the hill and caught sight of the ocean and the islands and in the evening as I took out the trash and saw the sunset so garish that if it were a painting, you'd make fun of the artist.


I came across this poem because recently I had the great good fortune of having the opportunity to think about how to bring the American Romantics into certain high school classrooms in a certain state (intentional vagueness required by both professional courtesy and the stipulations of the non-disclosure agreement that is part of my contract).


For a week, I read Emerson, Lake, and Palmer Emerson ("To believe your own thought, to believe that what is true for you in your private heart is true for all men, — that is genius"), Whitman, Dickinson, Dunbar. Leaving out Thoreau wasn't intentional; it's just that I started with Emerson, but once I was in the thick of the poetry, decided to stay there and tackle the essays another time.


This work was pure pleasure. Poetry and grade 11 are a match made in heaven. All those emotions, for one thing. And there's a poet for everyone. Baudelaire was the original Emo:

Dark one, I am torn
By your savage ways,
Then, soft as the moon, your gaze
Sees my tortured heart reborn.

                                      --from "Afternoon Song"

Not the least of the pleasure was my thinking about the students. Maybe there would be one or two who for whom these poems might be a signpost to the onramp to the highway that leads to this gorgeous world of poetry and self-knowledge.


Last week, I went to hear a friend and fellow alumnus from the College of Creative Studies at UCSB  speak about literature and read from his writing. He told the students about how, when he graduated, he had this sense of having a gift, a talent, that was bigger than he knew what to do with.


It was true of all of us, I think. I'll go further and say that it may be true of everyone, but maybe everyone is not lucky enough to understand that he has a gift, or not lucky enough to land in a place where he has the room and space and encouragement to find and exercise his gift.


My daughters sweetly and patiently listen when I talk about my work and endlessly quote from my readings. They have more than a passing familiarity with Emerson by now. Erin, my eldest, showed me what her social studies textbook says about Emerson and Thoreau.


The targeted standard is to read the writings of the American Transcendentalists. The means employed by the textbook writer to address this standard was to write one spare paragraph about these two in which Thoreau is described as someone who was jailed because he refused to pay a one-dollar federal tax and Emerson is described as someone who didn't want to go to jail and so he paid the tax. That's it. Thus summing up the philosophies and values of American Transcendentalism. There was no context, no real biographical information, there were no excerpts from the essays.

Why in the name of all that is holy would a writer (or a publisher, as surely it was not solely the writer's decision--as a writer, I often bump into the decisions of editors and publishers) pass up the opportunity to offer more?

In the immortal words of Emily Dickinson (see below), the brain is wider than the sky. Even at 14. I'm sure their brains could not only accommodate bushels, nay, truckloads of real information, rather than a dismissive tag line. Why not offer the miracles and let the students sort them out?

CXXVI THE BRAIN is wider than the sky, For, put them side by side, The one the other will include With ease, and you beside. The brain is deeper than the sea, For, hold them, blue to blue, The one the other will absorb, As sponges, buckets do. The brain is just the weight of God, For, lift them, pound for pound, And they will differ, if they do, As syllable from sound.

UPDATE: Fixed link to College of Creative Studies at UCSB.

Big Wheel Keeps on Turning

This just in, from Frank Brockmann, Center Point empresario and fellow quality crusader:

President Barack Obama announced last week that 10 states will be exempt from the requirements of the highly-criticized No Child Left Behind legislation. In exchange, those states will have to agree to a series of reforms. But some experts say the law should be scrapped completely for models that don't rely on standardized tests.

Interesting, because last night I was reading about the rise and fall of the Roman Empire California Learning Assessment System. CLAS was not the first--and surely will not be the last--testing program to be felled by public controversy. Among the many objections to CLAS were to the reading passages (two of Alice Walker's stories were excised after complaints that "Am I Blue" promoted vegetarianism, and "Roselily"was anti-religion), to the writing prompts (which were considered invasive), and to the test questions themselves (too subjective and based in emotion, according to critics).

When I began my career in hand-scoring, CLAS was one of my first projects, so I did at one time have more than a passing familiarity with the tests and how students responded to them. In general, I liked the questions (what did I know? I hadn't any content development experience at that point), although scoring did present a problem.

In some parts of the test, students were encouraged to take notes--which were called "marginalia"-- and for some questions, were offered various options as to how to respond: they could draw a picture, for example. Anyone who's ever been presented with a drawing by a small child understands the obstacle in scoring there:

You: Oh, what a lovely dinosaur.
Child: It's not a dinosaur.
You: No?
Child: No.
[pause]
You: What is it, love?
Child: It's a manatee.
You: Oh, yes, of course it is.

I see what the developers and supporters of CLAS were attempting to accomplish, and the goal is a laudable one: to lure students into engaging with authentic literature and then to welcome their genuine, individual responses to what they read. However, in retrospect, these goals might be more readily achieved at the classroom level, through both instruction and formative assessment (and by "formative assessment," I really mean all those teacherly techniques for paying attention to one's students, not the administration of a series of badly-written multiple-choice quizzes), than through a standardized testing program.

The point is really to define clearly one's purpose, and then create the assessment that will best serve that purpose, as "the man without a purpose is like a ship without a rudder," in the immortal words of Thomas Carlyle.

Big wheel keeps on turning. We went from bubble-in to performance assessment, back to bubble-in, and now we're talking about performance assessment as if it were the hot new thing, never before attempted.

Not that we shouldn't move in that direction--I like that direction very much--but if we do, there should be a moment of planning and taking stock (For which of you, intending to build a tower, sitteth not down first and counteth the cost, whether he have sufficient to finish it?), a gathering of the elders for the harvest of wise counsel, before there is a marshaling of forces. What is our purpose? What will we gain, what will we lose, is the loss worth the gain? How do we prepare teachers for this role?

What do you think?

In Defense of Quality

If you cannot learn to love real art, at least learn to hate sham art.

This, from William Morris.

By “real art,” let’s say James Lesesne Wells, for example, and by “sham art,” let’s say Thomas Kinkade. 

I’d like to apply this sentiment to the work of writing, and more specifically, to the work of test content development. Quite frankly, I’m mystified (a phrase I borrow from an assistant district attorney with whom I used to work when I was on a very different career path, and who was in the habit of using this phrase to sharpen his tongue as he prepared to slice me up for having done something with which he disagreed) by not only the deep and devastatingly obvious diminution of quality in test content in the past few years, but also by the failure of people in this silo of the industry to recognize this trend.

Quite frankly, it breaks my heart.  As silly as it sounds. But when you love, you expose yourself to the risk of heartbreak. Again, I turn to William Morris, who said, “Give me love and work – these two only.” I’ve been doing this work for 19 years now; though I got into it thinking it was a temporary rope to keep me out of the quicksand until I found my magic circle niche place on solid ground, I think we can all agree it’s become a long-term relationship.

If this lack of quality trend were limited to newcomers to the business, we could propose that them entry-level young’uns [*sigh*] are poorly educated and ill equipped to express themselves except via texting, which you can certainly see in their editorial comments (which are lamentably rich in acronyms, emoticons, and which betray an unfortunate juvenile fondness for excess punctuation and using all caps in directions, which cannot help but set one’s back up, however patient one might be, and anyone who knows me knows that overly patient I be not).

But no, all we content dev folk – ELA, math, science, and social studies, not one is immune, no, not one – have noticed, and we do talk about it, and the conversation and all the various repetitions and iterations of the same conversation bore and horrify us so that we are reduced to shaking our heads and turning our attention to some vision of an oasis, such as the cocktail that awaits the end of the day.

Back in the day, when I worked at Great Big Huge Test Publishing Company, I went to a mandatory training on root cause analysis. We used the fishbone chart. As trainings go, it was all right. Certainly better than the one at which I was accused of not doing my work and letting my teammates pick up my slack because I failed to participate in the assembling of a puzzle, which failure actually had a lot to do with my abysmally poor spatial intelligence and equally poor vision (since corrected through the wonders of Lasik surgery) and little to do with my work ethic, which, as it happens, is about as Puritan as a work ethic might be. You can take the girl out of the working class, but you can’t take the working class out of the girl. But I digress.

If we performed a root cause analysis on the wreckage of Good Ship Quality, what would we find?

To answer that we’d have to go back to the beginning. When I started as a content editor, I was dedicated to one project. That project was my one, my only, my all in all. It was the same for my co-workers. That was the early 90s. In most states, large-scale tests were restricted to reading, writing, and math, and were administered at three or four grades (usually something like 4, 6, 8, and 10, or 5, 7, and 11).

Five years later, it was a whole different and bigger but not necessarily better ballgame. More states were testing more grades, and NCLB loomed on the horizon. As a supervisor, I was responsible for five projects. No one on my team was solely dedicated to any one project; each person, from editor to supervisor, worked on several.

I had a meeting with my manager that went like this:

Manager: [peering at her clipboard] All right, so you have State V, State W, State X, and State Y.

Me: And State Z.

Manager: Oh, I forgot about Z. Right. State Z. [scribbles a note on her clipboard]

Me: What is the order of priority?

Manager: [pause] They’re all priorities.

Me: With five states, mistakes are going to be made. It’s impossible to supervise five projects of this scale. Which state is going to be the mistake state?

NCLB  hit like an atom bomb. 

Test publishing companies couldn’t handle the workload. Companies that had never done any testing smelled the money and jumped into the fray. All companies got hiring fever. By then, I was a content development manager hiring entry-level candidates at more than twice my starting salary as an editor (and did that ever sting, I tell you what).

But the equation for meeting a production deadline is

TIME + WORKERS = MEETING PRODUCTION DEADLINE

If you have less time, you need more workers. Fewer workers, you need more time. I am no math expert, but this equation I know.

Deadlines got more and more compressed, development cycles shrank, and everyone starting skipping steps. Real training gave way to on-the-job training, which really means sink-or-swim training. New-hires were handed the comprehensive binder containing lists of processes and procedures, which binders were relegated to shelves in cubicles because no one had time to read them. Early field tests were cast aside. Sometimes all field tests. There were fewer internal reviews. The few remaining reviews were performed by overworked and/or underexperienced staff—and you can actually determine which is which (and which is both) when you see the editorial feedback coming out of such reviews.[1]

Another significant factor may be a corollary to the Peter Principle. The most highly skilled, knowledgeable, and experienced line staff keep getting promoted to management, where they may be doing a fantastic job, but their spots are filled either by new hires or old hands who are left behind (how can I say this delicately? Their remaining behind may not always be by their own choice). Combine this with the absence of training, and it’s a chaos cocktail.

Not to mention the dependence on freelancers. Companies started laying people off and then rehiring them as subcontractors. For some, it’s a win-win—the company don’t have to pay your benefits, and you get to work at home in your pajamas—but it do mean there are a heck of a lot of people at their keyboards writing test questions who neither have experience in education nor in publishing, let alone assessment, which some of us choose to believe is both an art and a science.

There is value in enduring years of slogging through the entire publishing cycle from first draft through bluelines over and over and over again. There is value in having logged many hours in the company of small children struggling to read. There is value in meeting with what the industry calls the stakeholders—teachers, administrators, community leaders, DOE officials. There is value in stretching to accommodate the demands of the stakeholders. There is value in educating oneself about the history and practices of one’s profession. Those learn-to-play-the-piano-in-10-minutes books aside, there is no shortcut to attaining mastery in anything.

There are so many facets to what we do in assessment content development, and when one’s experience is restricted to one tiny mirrored triangle of the great big disco ball, well, that creates a problem because one hasn’t constructed a greater context which allows for greater meaning to inform and guide the work. When the work is simply writing questions for a paycheck and meaning goes out the window, the questions get lamer and lamer, by which I mean trivial, superficial, and plagued by error.

However, the purpose of identifying a problem is not to castigate wrong-doers, nor to enjoy that most basic human pleasure of being right, but to use such identification to find a solution.

The answers are probably as clear to you as they are to me:

• 1.     Only hire content developers (freelance or in-house, I have no axe to grind here) who either have a proven track record of providing high-quality work or who have the capability (combination of education, writing skills, content area expertise, intelligence, creativity, and persnicketiness) to learn how to do the work well
• 2.     Provide not only adequate but excellent training
• 3.     Employ senior content development personnel [*ahem* not naming any names] to review items and provide specific instructional feedback to writers
• 4.     Budget sufficient time and money for the given project

Well, there you go. There’s nothing new under the sun, in the immortal words of King Solomon. 



2/23/12 UPDATE: This just landed in my inbox, posted by Erik Robelen at Curriculum Matters at EdWeek:

The “new normal” in educational publishing is “a severe lack of oversight in the quality of curriculum being produced” and a “frightening apathy” to do anything about it.  Keeghan’s piece, “Afraid of Your Child’s Math Textbook? You Should Be” is a jeremiad.  It does for textbook publishing what The Jungle did for the meatpacking industry. 

Keeghan paints a bleak and dispiriting picture of a business gutted by mergers, competition for fewer available dollars, and an increased focus on sales and marketing at the expense of producing quality products.  Materials rushed to market at breakneck speed are “inherently, tragically flawed.”  Plus the pool of qualified writers and editors is drying up, and those doing the work “often don’t have the necessary skills or experience to produce a text worthy of the publisher’s marketing claims,” she writes.

 You will want to read Annie Keeghan's original blog post here at Open Salon.

---

[1] Overworked but highly experienced people skim text, which forces their brain to fill in the gaps. Which means erroneous assumptions and conclusions drawn from limited evidence and resulting in unnecessary, ill-advised edits. Subtleties or fine distinctions are impossible to detect when skimming. Underexperienced people often restrict their scope of what’s acceptable to their own narrow band of direct experience, and then reject what lay in the outer darkness of their ignorance. This is bad enough on its own, but they will then assume a pedantic tone and lecture the writer for having written items that exceed the demands of the specifications.  

Wrapping It Up

Another school year ending, while students and teachers rejoice, and we prepare for the anachronistic ritual of American summer, thus setting our children up for a lifetime of annual disappointment as working adults--unless they take up teaching.

Not to get on my hobbyhorse of increasing the school year--as much as I do believe that is the way, the truth, and the light. Instead, I'm thinking about all those adorable little savages, those incoming kindergartners, and I'm hoping that their parents have been talking and playing with them and reading to them and taking them to the library.

I met with an elementary school principal recently who described her incoming kindergartners of the last few years as bringing fewer skills and less knowledge to their first day of school than ever before. It would be depressing, if one's mind worked that way, but this principal is a powerhouse, a woman of firm and decisive action, and her mind is already working toward a solution: meetings with parents of preschoolers to let the parents know what they can do to prepare their children for kindergarten.

This isn't something everyone has to think about. Not thinking about it is a luxury afforded to parents of a certain education and socioeconomic level--the ones whose way of living grants their children automatic advantages. Lots of books in the home, trips to libraries and museums, music and dance classes, even just the time for lots of chatter and imaginative play. Crayons and paper and glue and let's not leave out the best of all, lots and lots of glitter.

You Gets What You Pays For

A friend forwarded this ad from a freelance job website:

Looking for 200 multiple choice trivia questions on the subject of Easter. Questions must be divided into ten different topics, each with an easy and hard section. All sections must have the same number of questions. Looking to spend no more than $30 on this, and the questions must be completed within 2 days. All i require is a spreadsheet, with ten different sheets. There should be four possible answers for each question, with the correct one in bold. I am only looking for people who speak fluent English. Please send two samples questions if you are interested. Copyright of all questions will belong to me and you may not reproduce them elsewhere.[Ed. sic, sic, sic.]

Though I often confess that numbers and me, we're not the best of friends, sort of like cranky neighbors who give each other a grudging nod once in a while when we catch sight of each other taking out the trash, let us do the math: 200 test questions in 2 days for $30. Which means, assuming an 8 hour work day (8 hours for work! 8 hours for sleep! And 8 hours for what we will!), that the rate of production will be 4.8 minutes per question at the whopping rate of pay of $1.88 per hour. Don't spend it all in one place.

My friend and I decided that a zero or two must have been left off the $30. But I do have a tiny little voice nagging doubt. Could it be?

Fun with Point Biserials

A few weeks ago, I was spending a lot of time analyzing test data, specifically, the p values and pt. biserials of a set of English language arts tests for grades 3 through 11, in order to determine What Went Wrong.

In the preponderance of cases, of course, nothing went wrong. The items performed more or less as expected, with the correlations one might expect--i.e., high-achieving students got the easy questions and most of the hard questions right, while struggling students may have gotten the easy questions right, but pretty much got the hard questions wrong. And it follows that no wrong answer in these solid items attracted more students than the right answer.

But there were items with wacky data, items for which the high-achieving students picked wrong answers, or items with wrong answers that lured a higher number of students, leaving the right answer feeling like an awkward wallflower in a darkened gym at the middle school dance. My task was to review these items and figure out the big why. It is a testament to my thorough and absolute geekiness that I LOVE DOING THIS WORK. I could do it all day long, every day. Oh, my goodness. More fun than a barrel of monkeys. It was such a pleasure that I found it difficult to tear myself away at the end of the day (and honesty compels me to add that I did sneak back to it at night after my daughters were asleep).

Why, you may ask. Oh, for so many reasons! One being that it is fun to play detective, to deconstruct an item by conducting an investigative inquiry which concludes in determining the most likely source of the trouble, whether it be a stem of staggering verbosity, or a fundamental unsoundness in the premise of the item, or a bad practice (e.g., attempting to assess two or more skills with one item). It's like picking up a big tangled knotted mess of yarn and, starting with one end, delicately unraveling the knots and twists and then rolling the yarn back into a nice orderly ball. But with your brain. How often do we really have the opportunity to use our brains in this way, and truly, what can possibly be more satisfying that solving a problem?

Because assessment content development, as its own universe, is governed by its own rules, and--to me, in my ridiculous and relentless geekiness--these rules have a sort of simple elegance, the beauty of rigor and orderliness. (Were I to wax biblical on the matter, I would start talking about the necessity for doing things decently and in order.) So to use the data as a mirror to reflect the soundness (or unsoundness) of the item, diagnose the trouble, and then find a means of repair requires the use of a range of tools that include a knowledge of the rules (and knowing when it is acceptable to bend them) and of how students might interpret an item (which is not always how a test developer intends) and of the content area itself. And it also requires a bit of intuition, that ability to sense your way in the dark that just comes from thoroughly knowing something, in the same manner you could navigate your home with the lights off because you know where the couch is and where stands that big coffee table with the treacherously sharp corners.