Showing posts with label test preparation. Show all posts
Showing posts with label test preparation. Show all posts

Thursday, January 24, 2013

Why Literature Matters: Part 1, or What Trollope Says About Assessment

I'm not sure I want to know anyone in this industry who hasn't at some time or another experienced a crisis of conscience. Humans seek meaning in our work, and unless we are psychopaths, we generally prefer doing good rather than evil. When we have a choice that doesn't much inconvenience us.

Why do our consciences rail at us during the dark nights of our souls? You know the answer. You've thought the answer yourself, or you've even asked me the question to elicit the answer, whether you be a friend, family member, or complete stranger sitting next to me on Southwest flight 1207 out of LAX.

If you are a parent or a teacher or a friend of a parent or of a teacher, or if you know and care about a child or if you ever think about what you read in the newspaper and magazines or consider what you see on the news, you know something about K-12 assessment and you probably have opinions about it, opinions that--please forgive the plain speaking, and this is through no fault of your own--are most likely ill-informed. Being as that there are few industries so shrouded in mystery and that excite so little enthusiasm and generate so little va va voom, so little sparkle as that of educational assessment publishing.

However, you probably understand that any conscious (let alone conscientious) person who creates tests for a living has concerns about the quality of the tests, the purposes for which they are used, and the consequences when they are put to use.

A potential salve to our consciences lies in 10 Reasons Why Assessments Make the World a Better Place (compiled by John Kleeman at Questionmark and sent to me by my colleague Frank Brockmann of CenterPoint--you see, we all three of us are soldiers in the same militia).

And so I, dutiful foot soldier that I am, armed with the twin purposes of promoting greater understanding of assessment and its implementation and consequences, and of reminding all of us why the study of literature is crucial to developing higher level thinking skills (and character!), will, with the aid of my betters, review these 10 Reasons Why Assessments Make the World a Better Place.

Today, let's tackle Reason #1: Assessments give equality of opportunity.

The idea that assessment ensures equality of opportunity is one that Victorian writer (and career civil servant) Anthony Trollope treats humorously in The Three Clerks (Chapter 3: The Internal Navigation) and seriously in his autobiography:

Italics mine.

SAT tutoring, anyone? How much does that cost these days, anyway? Who is most able to afford SAT tutoring? How does that disparity affect this playing field that the test is supposed to be leveling?

And here is what Trollope has to say about similar kinds of narrowly targeted test preparation:

Italics mine, yet again.

What Trollope says about himself is that he would certainly have performed poorly on any such public examination, but if he had been rejected based on his poor performance, the government would have just as certainly lost "a valuable public servant." That he was valuable is incontestable; every biographical (and autobiographical) account I've read indicates that Trollope applied himself to his career as devotedly, industriously, and with as much competence as he did to his writing. (You may not know that Trollope wrote a gazillion books, and that he rose every morning at 5 dark-thirty to write for two hours before he began his work day at his day job.)

Which ultimately means that Reason #1 doesn't really reassure me that I am making the world a better place, one test question at a time.

(In addition, a highly inconvenient truth of this industry is that the soundness of the test cannot always be taken for granted, and an unsound test, a test that is constructed in absence of best practices, should not be relied upon to produce data that is itself reliable. I predict this truth will apply to all ten reasons.)

Monday, February 20, 2012

File Under: All Roads Lead to Rome, Cross-Referenced to Mandatory Reading

The book every K-12 content developer--assessment and curriculum--should read is Tested: One American School Struggles to Make the Grade, which was recommended to me by a colleague and likeminded comrade in quality assessment content development, Carmen, a senior level genius expert at Anonymous Testing Company.


To say that the tests administered yearly at grades 3-8 (inclusive) are high stakes cannot possibly begin to convey what this means for students, teachers, and school administrators and how NCLB has transformed the educational landscape. The flames are licking at their feet every minute of every day.


And even when we who are informed are talking about this, and about how awful it is that teachers must teach to the tests and that the assessments are driving the curricula--it's one thing to talk about it, and it's another thing to experience it.  If you're not a student, teacher, principal, or parent, this book is the closest you can get to the fire.


Like the children in Tested, my daughters' skills and knowledge are assessed so frequently at school (and what is taught is often so narrowly focused--the algebra teacher actually labels each homework assignment with the assessable standard and tells parents that she does this so she will know whether students will answer those questions correctly on the state test) that I'm shocked by how little genuine instruction they actually receive and so I supplement their classroom instruction by offering my own reading, writing, social studies, and science lectures. Which may bore them nigh unto death, who knows, but I refuse to send them out into the world as little ignorami ignoramuses. (For math homework help, we turn to my friend and cohort and math content area genius expert Carrie Frech, who works at a major testing organization).


My daughters came home last week emitting little puffs of indignation over the latest district benchmark assessment. (They swiped it and brought it home to show me.) When I read it, I was horrified. It was a passage-dependent writing prompt. I didn't see a rubric, God knows what horrors hide behind that curtain, but I assume the responses were scored for reading and writing.


The story, a tedious adaptation of a folktale, was poorly written and at the fourth grade level (this, for an eighth grade gifted and talented program; my daughters are currently reading The Great Gatsby and Rebecca for their next book reports, and yet they're being assessed with text suitable for fourth grade?). What was there was was presented at an extremely literal level of understanding. Nothing in the story allowed for any genuine analysis of narrative elements nor interpretation of literary devices, and yet the writing prompt required the students to do just that. I don't know how they could. You can't make a pie out of one apple. The multiple-choice section (developed by a company relatively new to the game for whom I'd done some work a few years ago and by whose lack of understanding of test development at that time had shocked me) was no better. The girls told me that there was one question that was so nonsensical that, districtwide, the teachers' form of protest was simply to give all of their students the answer. Does anyone see any value whatsoever in the use of such an assessment tool?


When I did some work for this company a few years ago, I observed their inexperience with and lack of knowledge about assessment. Maybe things have changed since then, I don't know. What I do know is that this company--the same one that has little assessment background--doesn't perform any field testing of the test content and there is no data whatsoever to indicate that we can make any kind of accurate inferences about what students know and can do based on such poorly constructed assessments. But this company does a good job of selling, and the districts buy the dream fantasy idea that the products will 1) give teachers information about their students that will help them get students ready for the state test and 2) predict how students will perform on the state test. Neither claim is possible, particularly with assessments that violate the most basic quality standards.


All roads lead to Rome; it all comes back to the Quality Manifesto.


(And it must be said that certainly there is room for quality improvements in the classroom as well, and there was room even before NCLB. What passes for instruction in some classrooms horrifies me just as much as any quality train wreck I see in the assessment world. Two teachers at my daughters' school ROUTINELY play audio of the textbooks instead of teaching--this, in science, which I think we can all agree requires hands-on instruction; it's still bad in reading, but not quite so bad). One of these teachers also ROUTINELY spends twenty minutes or so of the fifty minute class period discussing her personal life to the captive audience of eighth-graders--they know all about her kids, her husband, her political views, her hobbies, her extended family, and her domestic habits. So do I. Unless someone is a friend, loved one, or crazy person celebrity, there's probably not much in her life you want to hear about for twenty minutes straight, and yet that is what this teacher subjects her students to instead of teaching them.)


UPDATE: Identified my book-readin' comrade by name. Thanks, Carmen!
UPDATE: Added a link.
UPDATE THE THIRD: Removed a link, anonymized an identity.

Friday, April 17, 2009

My Dream Project

Sometimes I amuse myself by thinking of fun little work projects I would like to do. My dream projects.

A few years ago, I read about a study in Florida which got me thinking about developing some kind of pre-testing curriculum. (Testing isn't going to go away. Nor should it, entirely. Whenever anyone waves the anti-testing flag in my face, I think about how none of us would like to the patient in an operation performed by a surgeon who had never undergone any tests of his medical knowledge. We wouldn't even want our cars repaired by technicians who hadn't ever been tested and certified. However, let us all agree there is room for reform in state and district testing, and leave it at that.)

So whenever we administer these tests, these tests on which some children score disproportionately lower than others, why not find better ways to prepare them? Not with test prep products, but with lesson plans that tell about the brain and how it works, and how we--we, all of us, all humans--gather and store and use information. Thereby making sure that the students understand that all humans have the same kinds of brains, brains that work the same way, and no one group has brains that are inherently better or less able to function than those of another group.

We could add a little writing exercise to the lesson plan:
ScienceDaily (Apr. 16, 2009) — In a follow-up to a 2006 study, a University of Colorado at Boulder researcher and his colleagues found that an in-class writing assignment designed to reinforce students' sense of identity and personal integrity increased the grade-point averages of African-American middle school students over a two-year period, and reduced the rate at which these students were held back or placed in remediation.
This follow-up just confirms the results of the 2006 study. This kind of writing exercise clearly has a positive effect on student performance. Why is it not common practice yet?