Wednesday, November 11, 2009

Fun with Point Biserials

A few weeks ago, I was spending a lot of time analyzing test data, specifically, the p values and pt. biserials of a set of English language arts tests for grades 3 through 11, in order to determine What Went Wrong.

In the preponderance of cases, of course, nothing went wrong. The items performed more or less as expected, with the correlations one might expect--i.e., high-achieving students got the easy questions and most of the hard questions right, while struggling students may have gotten the easy questions right, but pretty much got the hard questions wrong. And it follows that no wrong answer in these solid items attracted more students than the right answer.

But there were items with wacky data, items for which the high-achieving students picked wrong answers, or items with wrong answers that lured a higher number of students, leaving the right answer feeling like an awkward wallflower in a darkened gym at the middle school dance. My task was to review these items and figure out the big why. It is a testament to my thorough and absolute geekiness that I LOVE DOING THIS WORK. I could do it all day long, every day. Oh, my goodness. More fun than a barrel of monkeys. It was such a pleasure that I found it difficult to tear myself away at the end of the day (and honesty compels me to add that I did sneak back to it at night after my daughters were asleep).

Why, you may ask. Oh, for so many reasons! One being that it is fun to play detective, to deconstruct an item by conducting an investigative inquiry which concludes in determining the most likely source of the trouble, whether it be a stem of staggering verbosity, or a fundamental unsoundness in the premise of the item, or a bad practice (e.g., attempting to assess two or more skills with one item). It's like picking up a big tangled knotted mess of yarn and, starting with one end, delicately unraveling the knots and twists and then rolling the yarn back into a nice orderly ball. But with your brain. How often do we really have the opportunity to use our brains in this way, and truly, what can possibly be more satisfying that solving a problem?

Because assessment content development, as its own universe, is governed by its own rules, and--to me, in my ridiculous and relentless geekiness--these rules have a sort of simple elegance, the beauty of rigor and orderliness. (Were I to wax biblical on the matter, I would start talking about the necessity for doing things decently and in order.) So to use the data as a mirror to reflect the soundness (or unsoundness) of the item, diagnose the trouble, and then find a means of repair requires the use of a range of tools that include a knowledge of the rules (and knowing when it is acceptable to bend them) and of how students might interpret an item (which is not always how a test developer intends) and of the content area itself. And it also requires a bit of intuition, that ability to sense your way in the dark that just comes from thoroughly knowing something, in the same manner you could navigate your home with the lights off because you know where the couch is and where stands that big coffee table with the treacherously sharp corners.

Wednesday, October 7, 2009

An Angry Little Toot from a Lone Brave Whistle-Blower

The books I am reading right now are: Love and Will by Rollo May, Aretha Franklin's autobiography Aretha: From These Roots, and Daniel Goleman's Vital Lies, Simple Truths: The Psychology of Self-Deception. And I just finished Making the Grades: My Misadventures in the Standardized Testing Industry.

There may not, at first glance, appear to be any common ground (in the immortal words of Rev. Jesse Jackson) among these, but I will argue that as humans, we take ourselves wherever we go, and in so doing, we drag along the burdens of either consciousness or self-deception. As we choose. Sometimes one, sometimes the other, as best meets our needs and best fits our capacities at the time, the question of existence often being the same as that faced by Oedipus, (as Rollo May says): how much self-awareness can a human being bear? One hopes--I hope--that one may travel an upward trajectory in which one increases one's awareness of oneself and of the world, a trajectory that leads to some higher plane in which we can learn to live with truth.

We take ourselves to work, where we sometimes must walk a tightrope between our values and the need to earn a living. Most of us must make some compromises, must sell ourselves in some way. Some compromises are small and meaningless, but others may put our very integrity at stake. This is the real story of Farley's book.

Though the book is purportedly about the testing and test publishing industries, it is just as much about Farley, who presents himself as a whistle-blower, though one might be forgiven for the mean-spirited thought that Farley sure did take his time in finding the whistle, being as he kept on collecting a paycheck from The Great Satans for many years. And that perhaps this delay lends a bit of tarnish to his credibility.

It must also be said that Farley does not appear to best advantage when he writes about how he copied other workers' scores to get out of doing the work himself, or about spending most of his workday surfing the Internet, or about his hand-rubbing glee in charging exorbitant fees as a consultant (though maybe my pointing out the latter is evidence of envy on my part, as I cannot help but wonder how he managed to pull this off, as I have been a consultant in this industry for 8 or 9 years, and though I do support my little family, our style of living cannot be described as high off even a tiny hog). (Not to mention the subtle sexism in Farley's thinking that is revealed in his writing. Look at how his view of women is first invariably filtered through the lens of whether or not he finds them attractive. In his mind, women--no matter how accomplished or intelligent--are reduced to decorative objects because of course, a woman's main value has to do with whether you enjoy gaping at her. Then consider the adjectives and nouns he uses when writing about women. He says he traveled with "a gaggle" of women. Oh, please. Yes, we of the feminine persuasion are all just clacking geese, you know how ladies love to gab. Sigh.)

Reading Farley's book raises as many questions about him as it does about the industry he intends to expose. If it were so chock-full of despicable practices, why did he remain there for 15 years? How did making such a sacrifice of his own values and beliefs affect him? What exactly was going on in his mind as he participated in these ethics violations? While working in that industry, what efforts did he make for reform?

It seems that Farley wants to rail against this industry-wide malfeasance without taking any responsibility for his own role in it, but as it do say in the Bible, one cannot touch pitch and not be defiled.

This is a dilemma in which you might say I have a deep and abiding interest. The spirit of full disclosure compels me to state that I worked at CTB McGraw-Hill from 1993 to 2001, since which time I have been a content development consultant for a variety of test publishers, school districts, and one state department of education. Having worked with most of the major test publishing companies, I can say that I have seen a good share of corporate culture, and the more I see, the more I am glad I work for myself. Anyone who has ever worked in a corporate setting is probably familiar with at least some of the horrors Farley describes. People who are like the walking dead, who are so eccentric and odd-mannered as to seem unemployable, catch-22 mandates handed down from upper-level management, meetings that seem designed to showcase pomposity and vanity and futility.

But the shenanigans of wrong-doing in hand-scoring that Farley reveals, the behind-the-scenes falsification of scores, the pressure to score in one direction or another as the wind from on high changes, demands from psychometricians to increase the number of scores at a given grade point level, hand-scorers who were the dregs of society--these I did not see. Pressure to work faster, for higher productivity, yes. Unreasonable demands, yes. Obsequious sucking up to district or state officials, yes, yes, yes. Lots of co-workers with their little quirks, oh, yes.

As there is in the world, there is much that could and should be improved in this industry. On all sides, and probably in every department in every test publishing company. If Farley says that what he describes in the book was his experience, then that was his experience, I will not dispute that, though my own experience has been different. I agree completely that tests today are being used for purposes for which they should not be used. I agree completely that there is more to learning than can be measured by a paper and pencil test (or a keyboard test). I agree completely that one of the unintended consequences of the No Child Left Behind legislation has been the unleashing of unprincipled money-sniffing dogs into the industry (no offense to literal dogs), and the muck they try to pass off as genuine content--well! I have seen some awful terrible bad no-good things, is what I am saying.

And yet, testing is never going to disappear. Nor should it. The example I always give when a stranger tries to hold my feet to the fire is whether you would want to undergo an invasive medical procedure at the hands of a surgeon who had never submitted to (let alone achieved a passing grade from) any kind of examination as to his or her knowledge and skill. Let's face it, we don't even want to take our cars to mechanics who are not certified in some manner.

See, we can construct this evil villain testing empire, we can make paper dragon cut-outs all we want, but how does that effect real change? What about starting where we are? For myself, I find that much of my work has been coming more from curriculum the last few years, partly because these particular clients are just plain charming and nice to work with, but partly because, given the choice, I would rather be working on the side of remediation and intervention. Not that I have stopped my assessment work. However, if I ever do feel about it the way Farley did--if it stole from me my integrity--I hope that I would not hesitate to leave it behind. I can't really know what I would do unless I find myself in that situation. Some sleepless nights would result, I am sure. Rollo May says that fate plus guilt equals no rest for the wicked. (My paraphrase.)

Poppycock, Folderal, Nonsense

. . . in the immortal words of Todd Farley.

About a week ago, someone sent me a link to an Op-Ed piece in the New York Times by Todd Farley, author of Making the Grades: My Misadventures in the Standardized Testing Industry.

Farley's experiences aren't unique. Like Farley, I am a writer who sort of fell into the test publishing industry by accident. Like Farley, I stayed in the industry long after I thought I would have gone on to what I thought would be my real career of writing novels or screenplays or something, anything.

Both of us started our careers in hand-scoring, so hand-scoring is what I will talk about, specifically the hand-scoring of open-ended test questions. Multiple-choice questions are simple to score, because there is only one correct answer. All multiple-choice test questions are machine-scored. The answer sheets or test booklets are scanned, the answer choices verified by machine, and the scores are then computer-generated. Sometimes there are mistakes in the programming that must be corrected, for example, the correct answer to a given question was actually C but was identified somewhere along the line as A. Sometimes there are mistakes with a student's name or identification number that lead to a mistaken score. Sometimes--and this happened with my daughter's third-grade California STAR testbook--the testbook or answer sheet has juice spilled all over it, and so a false score may be generated. Where humans are involved, there will be some error somewhere, it is unavoidable, let us simply endeavor to put checks in place to catch the errors and processes to correct them.

The scoring of open-ended questions is a horse of a whole nother color. By its nature, there must be some subjectivity. In support of standardization are an array of tools that include a scoring guide or rubric, sample student responses at each score-point-level, and anchor papers and rangefinders. A rubric lists the characteristics of the response at each score point, a sample response gives an example of what kind of response is expected, an anchor is a student response that embodies the score point level, and rangefinders show what may be expected at the high, middle, and low ends of the spectrum within a score point level.

It sounds like a complicated process, and it is. And it's not without its ridiculous moments. And I have to say that though I found much about handscoring interesting, the work itself was tedious and the routine unbearable. But it's not the Orwellian circus of nonsense Farley describes. Or maybe it is at the company where Farley worked; it wasn't at CTB McGraw-Hill when I worked in hand-scoring there.

I am only about a quarter into the book, so maybe there will be some sort of Aristotelian discovery on Farley's part. At this point, he sounds like one of the disgruntled hand-scorers, and there were some of those, people who just never got it, never were able to internalize the scoring criteria and constraints, the ones whose scores had to be checked and re-checked so often that eventually they were let go. He says that he failed to qualify as a scorer for a writing test, which does make one wonder whether this type of work simply was not a good fit for him. Not that I can vouch for what happened at Pearson, as I've not worked there.

I will also say that--although I do not at all see myself as a flag-waver for the test publishing industry, and that I have my own strong feelings about the mis-use of tests and what seems to me to be an abuse of tests and how they are used and what they symbolize and how the data are manipulated--sitting in the mocking judgment seat is generally easy to do. I have plenty of ridiculous stories of my own. We humans are ridiculous, it's in our nature, and thank God that we are, it makes the world so much more entertaining.

And this book is just that--entertainment, a joke that is masked as an indictment of the industry. For myself, I'd be a lot more interested in a thoughtful exploration of the subject, one that takes into account the need for measurement in teaching, and the demand for standardization (because that seems to be the only way to ensure any kind of fairness or equity), and how we could possibly balance these kinds of standardized measurements with classroom performance and evaluations from teachers.

CORRECTION: I mean "Folderol." Geez. And to think I won first place in the 8th grade spelling bee. What did I tell you? Human error.

Wednesday, June 10, 2009

Why American Kids Fall Behind

Or maybe we should say, "Why American kids who are poor fall behind." There are many reasons, including the morally reprehensible way schools are funded, which inevitably leads to segregation by class and race and socioeconomic status. (Which adds up to the same thing, really.)

Yesterday I was reading Outliers by Malcolm Gladwell, so I was thinking about how short the American school year is. It makes no sense that Americans cling to long summer breaks for children. There used to be a reason for a longer summer vacation. In an agricultural economy, those children are needed to help on the farm. However, I think we can all agree that that time is long gone for most Americans.

In most other countries, children have a longer day at school, and they attend school for more days each year:
The accumulating data on comparative education, itself a relatively new preoccupation of policy specialists, point up two trends. First, compared with their peers in Asian and European countries, American students stand out for how little they work. Second, compared with Asians and Europeans, American students stand out for how poorly they do.
As to the first: consider a list, garnered from a variety of sources, of the varying number of days in a standard school year. This list was hard to put together--which tells us something about the neglect of this subject in U.S. educational circles.
Japan 243
New Zealand 190
West Germany 266-240
South Korea 220
British Columbia 185
Israel 216
France 185
Luxembourg 216
Ontario 185
Soviet Union 211
Ireland 184
Netherlands 200
New Brunswick 182
Scotland 200
Quebec 180
Thailand 200
Spain 180
Hong Kong 195
Sweden 180
England/Wales 192
United States 180
Hungary 192
French Belgium 175
Switzerland 191
Flemish Belgium 160
Finland 190
In the United States, this difference does not handicap children of affluent parents, as these children continue throughout the summer with their tutoring and piano lessons and family trips and online learning and all that. But what about kids whose learning is confined to a school setting?

What happens is that they lose some of the gains they made in the previous school year. When they return to school in the fall, teachers must waste time in review. Some don't, and then kids fall behind. The behinder they get, the behinder they get. Eventually, they get tired or overwhelmed by failure, and then they drop out. The drop-out rates for Hispanic and Black teen-agers is disproportionately high, which makes sense if you consider the disproportionate number of Hispanic and Black children living in poverty:

Children represent a disproportionate share of the poor in the United States; they are 25 percent of the total population, but 35 percent of the poor population. In 2007, 13.3 million children, or 17.4 percent, were poor. The poverty rate for children also varies substantially by race and Hispanic origin, as shown in the table below[4].

Children Under 18 Living in Poverty, 2007

All children under 18
White only, non-Hispanic

Here's a school where students spend more time in school. From ASCD.

2/22/12 UPDATE: Reportage from the NEA on longer school days.

Friday, April 17, 2009

My Dream Project

Sometimes I amuse myself by thinking of fun little work projects I would like to do. My dream projects.

A few years ago, I read about a study in Florida which got me thinking about developing some kind of pre-testing curriculum. (Testing isn't going to go away. Nor should it, entirely. Whenever anyone waves the anti-testing flag in my face, I think about how none of us would like to the patient in an operation performed by a surgeon who had never undergone any tests of his medical knowledge. We wouldn't even want our cars repaired by technicians who hadn't ever been tested and certified. However, let us all agree there is room for reform in state and district testing, and leave it at that.)

So whenever we administer these tests, these tests on which some children score disproportionately lower than others, why not find better ways to prepare them? Not with test prep products, but with lesson plans that tell about the brain and how it works, and how we--we, all of us, all humans--gather and store and use information. Thereby making sure that the students understand that all humans have the same kinds of brains, brains that work the same way, and no one group has brains that are inherently better or less able to function than those of another group.

We could add a little writing exercise to the lesson plan:
ScienceDaily (Apr. 16, 2009) — In a follow-up to a 2006 study, a University of Colorado at Boulder researcher and his colleagues found that an in-class writing assignment designed to reinforce students' sense of identity and personal integrity increased the grade-point averages of African-American middle school students over a two-year period, and reduced the rate at which these students were held back or placed in remediation.
This follow-up just confirms the results of the 2006 study. This kind of writing exercise clearly has a positive effect on student performance. Why is it not common practice yet?

Wednesday, April 8, 2009

Back to the Classroom

If anyone has a little extra time, that time would be well spent volunteering in an elementary classroom:
A program that uses older volunteers as tutors has significantly improved the reading skills of students in the early grades, according to a study released today.
. . .
The report found that the program had “statistically significant and substantively important” effects on the youngsters’ reading skills, as measured by standardized tests and teacher evaluations.
Speaking of the classroom, I may be teaching again soon. I'm applying for a position as a part-time lecturer at a local community college. It'll still be business as usual as Inkspot--you might say the teaching is for fun, or for an opportunity to volunteer in my own little way.

P.S. Some information to support literacy volunteering at the front end:

"There is not a lot of causal evidence that specifically says people with educational skills won't commit crimes, but there is definitely a strong correlation between educational ability and staying out of prison," said Peter Leone, a correctional education expert at the University of Maryland.

A comprehensive study by the Washington State Institute for Public Policy, the research arm of the Washington Legislature, found that general education programs reduced the recidivism rate by 7 percent and vocational programs by 9 percent, among the best records of in-prison programs.

The academic and vocational programs cost the state about $1,000 a year per inmate but, the study concluded, vocational education produced a net benefit to the state of $13,738 per participant, and the educational programs $10,669 per inmate, in the form of lower crime rates, fewer victims and less criminal justice spending.

Tuesday, April 7, 2009

More than a Check

Sometimes I approach my work sort of like the way doctors approach theirs: Above all, do no harm.

Tests are hard enough for students without our setting up unnecessary roadblocks. I like to think about writing tests that are transparent--that give a straightforward view of student performance. No obstacles. Which means sticking to all the item writing rules and writing simply and clearly.

But last week, I received an email message that reminded me that some of the work that I do--writing curriculum and assessment materials for intervention with struggling readers-- actually helps kids in a significant way, and that was a very welcome message.

To be a tiny part of a second chance for a kid who's at risk for dropping out of school, to be one of many people working on materials that will help this child become a competent reader--maybe help this child experience some rare success in a setting where he has only failed before--to just do that much is to do good. It's important to feel that one's work has meaning.

Tuesday, March 31, 2009

Under New Leadership

I am optimistic about the possibility of change for the better. In Washington, D.C., there is movement afoot to reform the schools. Toward that end, the district is accepting applications for principal and assistant principal. Apply here.

Monday, March 30, 2009

Not Such Good News

From Connecticut:
Results from the 2008 ACT college admission test show that many Connecticut high school seniors are "appallingly" unprepared for college-level work, according to a state report.

Of the 8,159 students who took the test, 35 percent of white seniors, 18 percent of Hispanics and 9 percent of African Americans were ready for college-level work.

"It's an appalling figure to look at," said Frank W. Ridley, chairman of the Board of Governors for Higher Education. "Basically it says that, at the very best, only one-third of our students are succeeding." Ridley said he was disgusted by the low numbers, which were included in a state report about racial and ethnic diversity in higher education. He said the disparity in the scores of whites and minorities underscores the state's struggle to close the stubborn "achievement gap."
Ridley says that the scores mean that "only one-third of our students are succeeding." Maybe the scores indicate dismal performance on the part of the schools. Or maybe the scores indicate that they are inadequately funded:
A Connecticut school finance lawsuit, filed more than four years ago with the high expectations of 12 towns that were challenging the way the state hands out education dollars, may not even make it to trial.
. . .
The case, Johnson v. Rowland, began in the spring of 1998, when the suit was filed on behalf of seven students and a coalition of the 12 towns. The plaintiffs have contended that the state has not contributed its fair share of education funding, and that poorer communities, like those that sued, are left with hefty tax burdens to pay for K-12 education.

Thursday, March 19, 2009

A School of My Own

A few years ago, someone approached me about starting a charter school. Although I was on board with the idea, quite frankly, I was horrified at the possibility of being the person who executed it. Me, a principal? I could not see it.

Still feel that way. I'm more of a supporting role actor. But I saw this article yesterday, via the ASCD newsletter, and it got me daydreaming about school.

ScienceDaily (Mar. 16, 2009) — Children exposed to a multi-year programme of music tuition involving training in increasingly complex rhythmic, tonal, and practical skills display superior cognitive performance in reading skills compared with their non-musically trained peers, according to a study published in the journal Psychology of Music.

It's old news that studying the arts helps children do better in other areas. And yet, and yet, we keep pushing art and music and literature aside in a sort of shortsighted devotion to Gradgrindian ideals.

Wouldn't it be great to teach students in a lively, interesting way that would make them want to learn? To include not only studies of content, but teach them how to do things--play music, build stuff, make art, grow plants, sew a placemat, knit a hat--so they could gain that beautiful confidence that always comes from competence? At this dream school, the curriculum would be integrated, let's say, so that kids would see that everything is connected, that there's math in music and biology in the garden and reading is everywhere, and so the skills and knowledge would not have this artificial demarcation, but would seep into real life.

P.S. Besides, pretty much any kind of intervention improves student achievement in reading. You can sit with a kid for thirty minutes a week and just give him your undivided attention, talk about basketball or video games or whatever, and his scores on reading tests will probably improve.

Monday, March 16, 2009

Incentive or Punishment?

I would rather think of it as incentive. But I like to look on the bright side.

You know that in 2006, Florida came out with a policy of linking teacher pay to student test scores. On the surface it might sound reasonable, but you have to roll back the carpet on this one to see the bugs. What about great teachers in underfunded schools? Great teachers in schools with high populations of second-language learners? Great teachers in schools where families are walking the razor edge of survival, where parents are working two jobs and can't help with homework, where some parents are MIA, having fallen prey to addiction, violence, or some other poison? See, no matter how great these teachers are, they need to grocery shop, too, and how many will be able to resist the call of being able to buy bread and cheese, and so they will migrate to the neighborhoods with higher-priced real estate. Once again, the kids who have the most need will end up with the worst teachers.

Now President Obama is talking about linking teacher pay to performance. Which makes a lot of people nervous. I am all for it, as long as the performance in question is that of teachers, not solely that of the students, and that the performance is measured in a fair way. We all of us in the (non-gummint) world are paid according to our performance. The man who shines shoes at the casino where I take my weekly UNLV class (everything in Las Vegas happens in a casino, it is just part of the local charm, I guess) makes more money because he does a good job (and is just plain pleasant to talk with); a server at a restaurant makes more money in tips because she is competent at her work. Writers can command a higher rate when they have established a solid reputation, and it's the same for consultants. There's no reason teachers should be exempt from expectations.

What I very much appreciate is the possibility of excellent teachers receiving higher pay. I've always thought if teachers could make a decent living, there'd be more excellent teachers. Last week, someone asked me why I didn't teach anymore (I taught a few community college classes in beginning and remedial composition, which I loved doing), and I said I couldn't afford the pay cut. I have to support my children, you see. And I would put money down--this being Vegas and all, I am picking up the local lingo--that I am not the only one.

That Time Again

The CCSSO Large Scale Assessment Conference is coming up: June 21-24. In Los Angeles, which is so convenient.

I believe I may have to take little field trips to see the stars on Hollywood Boulevard, and to LACMA. The Getty. If you have any other suggestions, please let me know.

Friday, March 13, 2009

Be Easy, Baby

. . . in the immortal but out of context words of Sharon Jones and the Dap Kings.

I often used to conduct item and passage review meetings with groups of teachers. I liked doing it, not only because I like being large and in charge. I also liked traveling to different states and meeting teachers from all over the country (and hearing their accents). I liked talking about test questions and reading test passages. I liked giving the teachers information about assessment, information that would help them in the classroom and for which they were invariably grateful. (I especially liked how teachers, when they are away from the classroom, always behave like students of the age they teach. The primary grade teachers are wide-eyed and very enthusiastic and sweet and need a lot of direction to keep them focused. The upper elementary grade teachers are lively and they talk out of turn and like to take breaks. The middle school teachers are sometimes a little difficult to manage. The high school teachers wear their sunglasses inside and sit toward the back of the room and slump down comfortably in their chairs. )

One of my first such review meetings was in French Lick, Indiana. A couple of hours in, I suggested we take a break, and encouraged the teachers to stand up and stretch and move around to refresh themselves, which suggestion inspired them to tease me about my wacky California-ishness.

It's been years, but now I get to say that I was right. And I do confess that I regrettably harbor a tiny fondness for being right.

P.S. Meditation is also beneficial to students, as is knitting. All three activities change the brain wave patterns:
Clinical studies over the years have shown the effectiveness of interventions on a wide range of medical problems caused or made worse by stress, such as hypertension, cardiac arrhythmias, pain, insomnia, allergies, repetitive stress injury and infertility, among many others. Practicing the relaxation response daily can boost the immune system and make one more resistant to the harmful effects of constant stress.
Good for all of us, but especially at-risk students, who may not have many resources to help them with the stress of surviving in this sometimes hostile world.

Tuesday, March 3, 2009

When Work Is Fun

One of my dear friends also happens to be Ms. Big Busy Very Important at one of these companies for which I sometimes work, and she gave me some hurry-up-we-need-it-now work last week. The task involved correlating English language arts standards created by one educational agency to standards created by another, and then aligning test questions to the first set of standards. Old hat. I've spent so much time keeping company with various state standards that there are some I know by heart.

Sometimes, though, it's the old hat that fits best. Once in the task, I was completely and blissfully absorbed by it. I had no concept of time passing, and the work felt almost effortless. Match, match, match, match, turn the page. Repeat. That would be flow, according to Mihály Csíkszentmihályi. I don't know how that happens, that one becomes so immersed in work that the work becomes a pleasure, but it is wonderful when it does. And yet, if you saw this big stack of test questions and the other big stack of papers with the printed state standards, it probably would look more like someone's final exam nightmare than a trip to the circus.

Also last week, I was in a class and the teacher asked what I did for a living. When I told him, he gave me that look that people often do give me when I tell them what I do and then asked, "Is it fun?"

That was a question I hadn't considered in a very long time. I had been working at a crazy pace the two weeks before, which hadn't been all that fun. Or I should say, would have been a lot more fun if I hadn't underestimated the amount of time the work would take me to do. (Because I like to live on the edge like that.) And of course, much of the last several months has been taken up with the culture shock of moving to a new state, and trying to begin putting my house in order, so I'd been working less than usual.

I thought about the teacher's question, and then told him that it was fun. And so once again, I had the opportunity to reflect on how fortunate I am, particularly these days, when not everyone gets to choose his occupation, and when even people who yearn to work are unable to find jobs.

The only other job I've ever liked so much as what I do now was making coffee drinks at an espresso bar. As dissimilar as it may seem. There is a common thread though, that of giving a client precisely what she wants and needs at that moment.


Today I got a shout-out asking about a practice to be employed in a classroom assessment, and it got me thinking about classroom assessments.

In this business, we think of classroom assessments as informal and low- to no-stakes, meaning that there will be no decisions made about student promotion/retention, and that there is no teacher/school/district accountability. There are probably some stakes for the students--they will receive a grade, which will contribute to an overall grade in the class, but the effect of this one grade on this one assessment is fairly minor in the great grand scheme of the universe.

Which is an excellent thing, as so few teachers know much about assessment, what skews an assessment, what practices should be avoided in assessment, which question constructions put up unnecessary obstacles for the test-takers, what kind of wording should be used. How do that happen, I wonder. Not to blame the teachers; the mystery is why teachers aren't generally required to take a class on test content development. Creating a sound assessment that accurately measures the targeted skills or knowledge requires knowing how to do it. You don't just get up from the couch one day and say, Hey, Ima build me a house without doing at least a tiny bit of research. But we don't know what we don't know.

Test content development is not difficult, but one does have to know what are the best practices in order to build a test that has at least some potential of giving a relatively fair and accurate measure of what a student knows and can do. Even if it is just in the classroom, even if the results only affect a tenth of the student's semester grade in one class.

Same thing at the DMV, you know. And those high-stakes tests could not possibly be legally defensible, constructed as they are in such a haphazard way, with all their outliers and overly attractive distractors and so on. Every single assessment professional who takes a DMV test cannot help but exclaim at the shoddy construction. What's so interesting to me about the DMV tests, though, is that even civilians recognize how bad the tests are, even if they cannot identify the reasons.

Wednesday, February 18, 2009

It Ain't Over Til It's Over

In 1971 in Orange County, Florida, 11 schools were essentially segregated, with a student population that was more than 80% Black. Today, 18 schools in the district are similarly segregated. One school, Orange Center Elementary, has a student population that is 99% Black.

Nearly 50 years ago, the NAACP filed suit against the district. Schools are still segregated. The suit is still open.

Thursday, February 5, 2009

Counting the Cost

We consultants sometimes agonize over our rates. How much should we charge for which tasks? Should we stick to our rates even when we don't have other work lined up? Is it ethical to charge different rates to different companies, or different rates for different tasks?

Questions like these used to plague me. Now, after 8 or 9 years of working for myself, I'm very clear about money; money conversations with clients are no different than any other conversations. To make money conversations easy requires integrity, clarity, and flexibility.

Not that you have to talk about having integrity. When you have it, it's part of all you do and say. For me, having integrity means being fair to my client and fair to myself. When I started my business, I decided upon an hourly rate. This decision was not made lightly. I had to consider my overhead expenses, and that I spend 10-20 hours a week on tasks for which I can't get paid--email, telephone calls, invoicing, banking, records, filing, and so on. After 6 years, I realized that my rate was inadequate--I could not afford to contribute to my retirement account, nor buy health insurance, nor could I take sick days or vacation days--and so I gave myself a raise. What is interesting is that I didn't lose any clients by so doing. I did have to refer some potential clients to other consultants who charge less. All of that having been said, I will also say that I often do free work. I rarely charge for telephone calls, ditto emails, and I often throw in this or that bit of work for free, especially for clients who have given me a lot of work over the years. It is very important to me that my clients know I will give them the best work I can do. Once, a client told me she was dissatisfied with my work, and I told her to pay me what she thought it was worth, as it was more important to me that she be satisfied than it was to receive the paycheck.

My hourly rate is at the higher end for tasks involved in content development. However, I can be flexible, given my schedule and my client's budget. Sometimes I will work for a lower rate for some simple tasks, especially if I have no other work lined up. Better to work at a lower rate than not to work at all. (Usually. There are some exceptions. Once I committed to a large project for which there were no written specifications--which is where the clarity comes in. I'd worked with the company before, though not with that particular editor. As we began mapping out the tasks, it became clear that the scope of work was much greater than I had understood from our earlier conversations. I told the editor I would have to charge more money for the additional work. The editor was unable to pay more, and we decided together to nullify my contract. These weren't pleasant conversations, although certainly there was no animosity; we both simply agreed that the project was not a good fit for what the editor called my "cost structure." It was the first and last time I've done that. You can imagine that now I am very, very careful to make sure I understand what is expected from me before I sign a contract.)

If a client's budget for a project cannot accommodate my rate, I can work with my client to find a mutually acceptable compromise. We might decide that my work will only involve design, specifications, training, and/or content review. Sometimes I can vend out lower-paying work to subcontractors. I do have to make sure I keep enough of a percentage to pay for my share of the work (assigning and explaining the work, and reviewing and editing it before delivering it to my client) and to pay the taxes on the income. I can also set a per project rate for clients at companies with a lower ceiling on hourly rates. This works best for clients with whom I've already established a foundation of mutual trust. I've found that most people want to be fair. I want to be fair, too. Once we know this about each other, it's a simple matter to figure out what they can afford for me to do.

If they can't afford everything they want from me, I can give a little. I often give a discount to clients who call me for work, as they save me money when I don't have to spend time fishing for work. I often give a discount to clients who offer me a contract for a large project, one that will last for several months or more, for the same reason. I am just about to start a venture that will take me through the first week of May; although I am working at a lower rate, the peace of mind that comes from having a steady income for the next few months is worth the sacrifice.

There are two more discounts I am sometimes inspired to give: one is the likability discount, and the other is the fun discount. You cannot overestimate the importance of likability, and sometimes the attraction of a fun project is so compelling that I'm willing to take a little pay cut.

Monday, January 26, 2009

Get a Job

In the ASCD newsletter, there is a posting for this job (complete job description is reprinted at the end of this post). It's a good job for someone with the required qualifications. Back in my CTB days, when I was a program manager, I worked with the Department of Defense Education Activity, and I must say that everyone there was lovely to work with. Anyway, I hate to see a good job going to waste, especially one with a reasonable salary and excellent benefits, so please pass this on to anyone who might be interested.

News of jobs do often come round my way, so I'll make it a habit to post'em as I get'em. If you hear of a job that's not right for you, but that is too good to let slip away, send it in.

Additional Duty Location Info: 1 vacancy - Arlington, VA
- Provides authoritative advice and guidance to DoDEA senior officials on all matters related to development, planning and implementation of the program areas to which assigned.
- Plans, organizes, and conducts a wide variety of studies and analyses designed to identify needs, develop justifications, and prepare implementation procedures for general and specific educational program requirements, usually of a critical nature.
- Formulates plans and provides managerial expertise regarding all aspects of staff operations.
- Provides program evaluation and analysis services for new, proposed or existing initiatives.
- Through subordinate supervisors and program managers who also exercise leadership responsibilities, plans, prioritizes, and prepares schedules for the completion of work.
- Identifies, facilitates, and monitors the resolution of problems, concerns, and issues involving education services provided by headquarters and a variety of other providers.
- Through subordinate supervisors and program managers who also exercise leadership responsibilities, plans, prioritizes, and prepares schedules for the completion of work.
- Develops, coordinates, and integrates educational programs, goals and other activities within and among the branches of the Education Directorate.
- In coordination with subordinates, develops goals, benchmarks, strategies, and a framework for the assigned program areas including those that support the National Education Goals and the Department of Education's Career pathways.
- Approved as a Contractor Officer's Representative (COR). Monitors the quality of program services to ensure compliance with professional quality assurance standards.
-Performs a full range of supervisory duties for both intermediate supervisors and non supervisory employees.

Tuesday, January 13, 2009

Why I Love Writing for NGEE

This weekend I had that experience that brings joy to all writers, that of seeing my name in print. There is my byline in nice big letters in the January issues of both National Geographic Explorer and National Geographic Extreme Explorer. I've written four articles for National Geographic Extreme Explorer, at which publication the editors are extremely likable. Extremely likable, and extremely good editors.

Where I come from, educational assessment and curriculum publishing, there's not a lot of editing (of stories and articles, anyway) for the sake of art and language. There is editing to get your word count, and editing to achieve a target readability, and editing to make sure the vocabulary is appropriate to the grade level. Before working with the NGEE editors, I'd been accustomed to writing my piece, turning it in, and getting paid. Once in a rare while, an editor might want this or that minor change to accommodate this or that project specification (or client preference or even whim), and of course I would make such changes with nary a grumble, but I was unused to revising according to direction.

Of course I revised on my own. I wrote and revised, wrote and revised, drank a glass of wine, slept on it, and revised again. Revising according to editorial direction is a whole nother matter. It takes a good editor, and by that I mean not just someone who edits well, but someone who is capable of clearly expressing his or her vision of the final piece, and making specific suggestions about what should be added, deleted, or changed to create that vision. Someone who can express and suggest in such a way that does not encourage the writer to curl up into a little ball of snottification and anxiety, as writers are sensitive souls and prone to weeping and moaning and gnashing of teeth.

And then the editor does even more magic before the issue goes to print, with the result that when I see my story, I hardly recognize it, because it is so much better than it was before.

Dear NGEE editors, I am your big fan.

UPDATE: Fixed a typo. If only I had the luxury of a good editor for ALL my writing. Blogs. Emails. Etc.

Friday, January 9, 2009

Green Grass

I like to say that self-employment has ruined me for corporate work. After having been the mistress of my domain for lo, these many years (8?), I doubt my ability to shed the pajamas, don the pantyhose, and kowtow to upper management (and there is always going to be a management Upper Than Thou, no matter how high the monkey climbs on the tree).

Like much of what I say, though, it's a joke. Sometimes I consider the prospect of employment with longing, say, when I am paying for my own doctor visits or when I get my 401k statement in the mail, or even when I feel unbearably lonely clicking away on the keyboard by myself all day.

The benefits of working for a company extend beyond the medical, dental, vision, and retirement plans. Consorting with others--especially smart, likable, cooperative others--can be very pleasant. Sharing responsibilities can ease one's mind. Being able to finish one's day and walk away from the office may be preferable to having one's office at home and knowing that there is always some task calling out to one.

And this discussion would be incomplete without an acknowledgment of the current economic landscape. Lay-offs have been legion through the holidays; unemployment rates are higher than they have been in decades. If one is fortunate enough to be offered a job when so many others are losing theirs, doesn't it seem akin to ingratitude to turn down the offer? And doesn't the offer appear all the more appealing?

It's worth thinking about, in any case, and not just in a "is the grass greener over there?" kind of way. In fact, at least two of my fellow free-lancers are thinking about going back through that gate, and if they do, I must say that I will think about it even more.

Besides, in terms of thinking about one's work, I like adopting a flexible approach. Be self-employed for a while, see how that works for you. Sure, take that job that calls out to you because you heed the siren's call of benefits/more money/steady paychecks/collaboration/more variety. Try it on, see how it feels, and if it doesn't work, the pajamas aren't going anywhere.

CORRECTION: I said that unemployment rates were higher than they had been in decades, but that was inaccurate. The unemployment rate is actually higher than it has been in 16 years.

Wednesday, January 7, 2009

But Is It Art?

A couple of years ago, I spent an entire month writing stories, articles, and poems for a shelf product for a company that shall remain nameless. What could be more fun than that? With the poems, I amused myself by choosing different forms in which to write. The children who later read the poems had no idea they were reading a villanelle or a poem in terza rima, but the discipline of a strict form gave me license to be more creative with both content and language.

There's a kind of beauty in rigor and discipline; you can see this in any creative endeavor, including item writing. I've seen items that were elegant in their simplicity and adherence to the demands of the form. Following all the rules doesn't make for a perfect item--even a flatly literal basic recall item may be written perfectly, and yet be a bad item. But it's impossible to be ignorant of (or ignore) the rules and write a good item.

That having been said, when you have completely mastered the rules, you may want to break one or two once in a while, for a specific purpose. The catch is, of course, that the flouting of the rule must be intentional and purposeful, or it just looks like what it is: a mistake.

Tuesday, January 6, 2009

The Poetry of Item Alignment

When I was an associate editor at CTB McGraw-Hill, back in the day, we editors were all a little grateful for the seasonal winter slowdown. It gave us time to catch our breath, file the stacks that had piled up during bluelines, and reacquaint ourselves with our co-workers. Now that I have been self-employed lo, these many years, I like to keep busy--whatever the time of year. Summer, winter, don't make no never mind to me. Idle hands are the devil's workshop.

Just before Christmas, I was doing a bit of writing, some for National Geographic Extreme Explorer, and some for ETS. (When the NGEE article comes out, I'll be sure to let you know. The other is, of course, confidential.) Then my attention was directed to alignments. Or correlations. The two terms are often used interchangeably; both mean identifying the standard (performance indicator, objective, skill, subskill) assessed or targeted by a specific item or task.

The best case scenario in content development is to write the item or task to a standard. That process is more creative, more organic, in the sense that the item or task may be developed to meet the demands of the standard. But many companies find themselves with a bank or pool of perfectly sound items, and to recycle these items for multiple projects is both efficient and cost-effective.

Alignments can be tricky. Sometimes items are shoehorned into standards that are an obvious bad fit. New aligners are especially prone to falling prey to aligning by key word, which is a big mistake. When aligning, it's imperative to keep in mind the spirit of the law, as opposed to the letter of the law. Think about the task and what the task requires that the student know or do, then review the standards (performance indicators, objectives, skills, subskills), and select the one that cleaves unto those knowledge and skill requirements. There may be more than one; there often is a lot of overlap. If no skill fits, better not to force the fit. How can a bad alignment result in meaningful measurement of skill or knowledge?

What I've noticed in reviewing others' alignments is that I can get a sense not only of the breadth and depth of the aligner's content knowledge, but of the aligner's intuitive feel for the content area (and for language use in general)--of the aligner's capacity for understanding subtle nuance. In a way, it really is like reading poetry.

Which may sound ridiculous, and it would be, if we were discussing a simple skill, such as "Use end punctuation correctly."