...is that you gotta have fun.
Every craft, art, skill, vocation--every discipline--comes with rules. Call them procedures, steps, processes--there's a set of knowledge that must be acquired and then applied over and over.
The path to mastery is
1. learn the rules
2. practice following the rules
3. make mistakes
4. get corrected
5. do it over
6. repeat eleventy gazillion times
Until you know enough to know when you can or should break a rule, when the value of such breakage exceeds the value of following the letter of the law. When you're really the master of a craft, art, skill, vocation, you're so at home in that world that you operate on a different plane--it's all intuitive. With experience and mastery come a set of tools that aren't available to beginners or even intermediates.
The most basic foundation for the rules is that you have to care about what you're doing, which to me means you gotta have fun. There must be something in there that's absorbing or it's not worth doing. Do something else. Do something you like, something you care enough about that you'll do a good job at it. You have to like what you do so that you care about it so that you learn to do a good job.
Speaking of twenty-year-olds, Louis C.K. says to just do the job you're getting paid for. [Might not want to play this at work. Taboo words. Don't play it if you're offended by profanity.]
But I don't mean to pick on writers. I believe that many would do better if they knew how to do better. What if the writers who provide okay work and the writers who provide almost okay work want to provide excellent work? Let's assume they have the solid foundation of caring about the quality of the writing and they want to make their editor's life easier. Now what?
There's no longer a now what in this industry. We've discussed this before. And before. There used to be a training path, but the shrubbery has overgrown the path.
Which is a long way of saying that I thought it would be good to write a series of posts to review some of the basic rules of item writing.
The test publishing industry is shrouded in mystery. First, because it's technical, and for lots of people, technical=boring. Whenever I begin to describe or explain what I do, I see all the light fade from the listener's eyes when his brain wanders off in search of sparkly entertainment. Then, because the tests are high-stakes, they are secure and confidential. Unfortunately, the security and confidentiality create an environment in which substandard work is passed off as acceptable. Who's to know? This is an industry that has incredibly high stakes for students and teachers, and yet it is completely unregulated. Think about that for a second.
The end users (those poor kids) don't know the difference. They just do what they're told. The teachers are hardly more sophisticated in this realm, and must trust their administrators, who must trust the district personnel, who must trust the state department of education, who must trust--the test publishing companies.
That whole chain might wind differently if assessment development occupied a place in teacher education--doesn't it seem strange that teachers are taught so little about what becomes of such vital importance to them and their students?
In any case, the plan is to review each of the basic rules, show what good items look like and what bad items look like, and explain what makes items good and bad. If even one teacher learned how to tell the difference between a good test item and a bad test item, that would be all to the good, and what I want is to increase the amount of good in the world.
7/23/13 10:01 p.m. UPDATE:
After publishing this post, I received a message from a writer. I realized I may have inadvertently hurt writers' feelings. That is so not my intention!
For the sake of brevity, I used a gross description. I intended that part of the post as a bridge to another matter. My intention is to address problems in the industry. My intention is not to pick on, discourage, nor cause pain to any writer, but to consider why this industry isn't hospitable to new writers and how we might be able to change that. The industry desperately needs talent, but erects insurmountable barriers for outsiders. This industry doesn't provide training. This industry relies on very few people to produce a high volume of work for pay that gets lower and lower.
It would take another blog post to explain the matter fully. The problems are becoming legion.
Instead, I deleted part of the post. To this writer and all of the writers I had the good fortune to work with during that project:
Thank you. I learned so much from that project. It was a pleasure to work with so many talented writers who produced so many passages that were charming, fun to read, interesting, and all kinds of fantastic. I appreciate your hard work and willingness to learn. I find it unfortunate that I didn't (and don't) have the luxury to train writers at a reasonable pace, nor sufficient time to coach writers on all of the inflexible demands of the industry. That does not imply a deficiency on the part of the writer. Every writer I worked with was capable of great writing. This industry might not be the best fit for some--and that might not be a bad thing. Or possibly one day I will have the luxury to be able to coach and mentor writers so that they can succeed in this market.
What I'm thinking about what I do, which includes but is not limited to all matters related to education, reading, writing, editing, content development, and curriculum and assessment design and implementation
Showing posts with label high-stakes. Show all posts
Showing posts with label high-stakes. Show all posts
Tuesday, July 23, 2013
Monday, February 20, 2012
File Under: All Roads Lead to Rome, Cross-Referenced to Mandatory Reading
The book every K-12 content developer--assessment and curriculum--should read is Tested: One American School Struggles to Make the Grade, which was recommended to me by a colleague and likeminded comrade in quality assessment content development, Carmen, a senior level genius expert at Anonymous Testing Company.
To say that the tests administered yearly at grades 3-8 (inclusive) are high stakes cannot possibly begin to convey what this means for students, teachers, and school administrators and how NCLB has transformed the educational landscape. The flames are licking at their feet every minute of every day.
And even when we who are informed are talking about this, and about how awful it is that teachers must teach to the tests and that the assessments are driving the curricula--it's one thing to talk about it, and it's another thing to experience it. If you're not a student, teacher, principal, or parent, this book is the closest you can get to the fire.
Like the children in Tested, my daughters' skills and knowledge are assessed so frequently at school (and what is taught is often so narrowly focused--the algebra teacher actually labels each homework assignment with the assessable standard and tells parents that she does this so she will know whether students will answer those questions correctly on the state test) that I'm shocked by how little genuine instruction they actually receive and so I supplement their classroom instruction by offering my own reading, writing, social studies, and science lectures. Which may bore them nigh unto death, who knows, but I refuse to send them out into the world as littleignorami ignoramuses. (For math homework help, we turn to my friend and cohort and math content area genius expert Carrie Frech, who works at a major testing organization).
My daughters came home last week emitting little puffs of indignation over the latest district benchmark assessment. (They swiped it and brought it home to show me.) When I read it, I was horrified. It was a passage-dependent writing prompt. I didn't see a rubric, God knows what horrors hide behind that curtain, but I assume the responses were scored for reading and writing.
The story, a tedious adaptation of a folktale, was poorly written and at the fourth grade level (this, for an eighth grade gifted and talented program; my daughters are currently reading The Great Gatsby and Rebecca for their next book reports, and yet they're being assessed with text suitable for fourth grade?). What was there was was presented at an extremely literal level of understanding. Nothing in the story allowed for any genuine analysis of narrative elements nor interpretation of literary devices, and yet the writing prompt required the students to do just that. I don't know how they could. You can't make a pie out of one apple. The multiple-choice section (developed by a company relatively new to the game for whom I'd done some work a few years ago and by whose lack of understanding of test development at that time had shocked me) was no better. The girls told me that there was one question that was so nonsensical that, districtwide, the teachers' form of protest was simply to give all of their students the answer. Does anyone see any value whatsoever in the use of such an assessment tool?
When I did some work for this company a few years ago, I observed their inexperience with and lack of knowledge about assessment. Maybe things have changed since then, I don't know. What I do know is that this company--the same one that has little assessment background--doesn't perform any field testing of the test content and there is no data whatsoever to indicate that we can make any kind of accurate inferences about what students know and can do based on such poorly constructed assessments. But this company does a good job of selling, and the districts buy thedream fantasy idea that the products will 1) give teachers information about their students that will help them get students ready for the state test and 2) predict how students will perform on the state test. Neither claim is possible, particularly with assessments that violate the most basic quality standards.
All roads lead to Rome; it all comes back to the Quality Manifesto.
(And it must be said that certainly there is room for quality improvements in the classroom as well, and there was room even before NCLB. What passes for instruction in some classrooms horrifies me just as much as any quality train wreck I see in the assessment world. Two teachers at my daughters' school ROUTINELY play audio of the textbooks instead of teaching--this, in science, which I think we can all agree requires hands-on instruction; it's still bad in reading, but not quite so bad). One of these teachers also ROUTINELY spends twenty minutes or so of the fifty minute class period discussing her personal life to the captive audience of eighth-graders--they know all about her kids, her husband, her political views, her hobbies, her extended family, and her domestic habits. So do I. Unless someone is a friend, loved one, orcrazy person celebrity, there's probably not much in her life you want to hear about for twenty minutes straight, and yet that is what this teacher subjects her students to instead of teaching them.)
UPDATE: Identified my book-readin' comrade by name. Thanks, Carmen!
UPDATE: Added a link.
UPDATE THE THIRD: Removed a link, anonymized an identity.
To say that the tests administered yearly at grades 3-8 (inclusive) are high stakes cannot possibly begin to convey what this means for students, teachers, and school administrators and how NCLB has transformed the educational landscape. The flames are licking at their feet every minute of every day.
And even when we who are informed are talking about this, and about how awful it is that teachers must teach to the tests and that the assessments are driving the curricula--it's one thing to talk about it, and it's another thing to experience it. If you're not a student, teacher, principal, or parent, this book is the closest you can get to the fire.
Like the children in Tested, my daughters' skills and knowledge are assessed so frequently at school (and what is taught is often so narrowly focused--the algebra teacher actually labels each homework assignment with the assessable standard and tells parents that she does this so she will know whether students will answer those questions correctly on the state test) that I'm shocked by how little genuine instruction they actually receive and so I supplement their classroom instruction by offering my own reading, writing, social studies, and science lectures. Which may bore them nigh unto death, who knows, but I refuse to send them out into the world as little
My daughters came home last week emitting little puffs of indignation over the latest district benchmark assessment. (They swiped it and brought it home to show me.) When I read it, I was horrified. It was a passage-dependent writing prompt. I didn't see a rubric, God knows what horrors hide behind that curtain, but I assume the responses were scored for reading and writing.
The story, a tedious adaptation of a folktale, was poorly written and at the fourth grade level (this, for an eighth grade gifted and talented program; my daughters are currently reading The Great Gatsby and Rebecca for their next book reports, and yet they're being assessed with text suitable for fourth grade?). What was there was was presented at an extremely literal level of understanding. Nothing in the story allowed for any genuine analysis of narrative elements nor interpretation of literary devices, and yet the writing prompt required the students to do just that. I don't know how they could. You can't make a pie out of one apple. The multiple-choice section (developed by a company relatively new to the game for whom I'd done some work a few years ago and by whose lack of understanding of test development at that time had shocked me) was no better. The girls told me that there was one question that was so nonsensical that, districtwide, the teachers' form of protest was simply to give all of their students the answer. Does anyone see any value whatsoever in the use of such an assessment tool?
When I did some work for this company a few years ago, I observed their inexperience with and lack of knowledge about assessment. Maybe things have changed since then, I don't know. What I do know is that this company--the same one that has little assessment background--doesn't perform any field testing of the test content and there is no data whatsoever to indicate that we can make any kind of accurate inferences about what students know and can do based on such poorly constructed assessments. But this company does a good job of selling, and the districts buy the
All roads lead to Rome; it all comes back to the Quality Manifesto.
(And it must be said that certainly there is room for quality improvements in the classroom as well, and there was room even before NCLB. What passes for instruction in some classrooms horrifies me just as much as any quality train wreck I see in the assessment world. Two teachers at my daughters' school ROUTINELY play audio of the textbooks instead of teaching--this, in science, which I think we can all agree requires hands-on instruction; it's still bad in reading, but not quite so bad). One of these teachers also ROUTINELY spends twenty minutes or so of the fifty minute class period discussing her personal life to the captive audience of eighth-graders--they know all about her kids, her husband, her political views, her hobbies, her extended family, and her domestic habits. So do I. Unless someone is a friend, loved one, or
UPDATE: Identified my book-readin' comrade by name. Thanks, Carmen!
UPDATE: Added a link.
UPDATE THE THIRD: Removed a link, anonymized an identity.
Subscribe to:
Posts (Atom)