Friday, January 25, 2013

Zip a Dee Doo Dah, or Go, Team!

As I work my way through the list of 10 Reasons Why Assessments Make the World a Better Place, I realize that this list is more of a grab bag of opinions festooned here and there by ribbons of fact than an argument resting on a solid platform constructed of actual information. (Not that there's anything wrong with that; what else are most blog posts than someone's opinion? I myself have got lots and lots of opinions.)

Which means I am silly for engaging in intellectual discourse about what is essentially an attempt to look on the bright side from deep in the trenches of a beleaguered and much-maligned profession. 

But once I've begun, I must onward go, silly or not. Let's address Reasons 2 through 10 from 10 Reasons Why Assessments Make the World a Better Place by John Kleeman:

2. Assessments make the world safer. 
Absolutely. No doubt about it. Certification and licensure are very reassuring, whether applied to doctors, nurses, mechanics, electrical engineers or dog trainers.

3. Assessments are the best way to measure knowledge, skills, and attitudes.
They're the only way, aren't they? If by "assessments," we mean "measurements," and assessments may include the use of observation and other tools beyond the isolated pen-and-paper or online experience.

4. Assessments are the cornerstone of learning
Not so sure about this one. Remember what Trollope said. How many modes of learning are taken into account? Which work best for which student?

And it would really depend on the assessment, how it was administered, how the results were interpreted and used, and whether instruction were subsequently guided by those results. 

5. Assessments  reduce forgetting. 
I believe I read something about this somewhere, but I didn't take a test on it and now I've quite forgotten what the article said, except for the bit about how you remember more if you are tested on it. 

6. Assessments are one of the few ways to be sure people really understand.
Subset of Reasons #3 and 4, and that is only if the assessments are solidly aligned with the curriculum.

7. Assessments give objective data. 
This is what they are intended to do. If the assessments are sound, constructed according to best practices, and free of obstacles such as cultural and other bias, we hope to obtain objective data after administering assessments.

8. Assessments define standards. 
Not so sure about this one, either. Often item review committees define and redefine standards. Sometimes individual item writers creatively define standards, and their work of a moment forms the template that is followed forever after. Assessments should define standards.

As an assessment content developer, I sometimes find myself stretching my brain until it snaps to find some logical way to target a skill that simply isn't assessable with a multiple-choice question. I know that it is not assessable, and yet I must do it, because that is the assignment. Am I defining the standard? I may very well be, but were I officially responsible for defining the standard, I'd approach it differently, perhaps starting off by working directly with groups of kindergarteners to obtain a baseline for what they actually are capable of doing and then burying myself in the library in order to see what people who spend their lives studying the developing minds of kindergarteners say about it.

9. Passing an assessment makes people feel good about themselves.
That's nice, isn't it.

Although I only feel good about myself when I absolutely and totally crush an assessment. 100% is what makes me feel happy, but we all have different standards, see Reason #8. A quiz will follow.

10. Online assessments give access for all.
I guess so, if a computer is made available to everyone.

And that's a wrap.

As ridiculous as I may seem in taking seriously what we can assume from the unbalanced and cheerleaderly perspective is most likely intended as marketing literature (using the term "literature" loosely because you know I am persnickety and Victorian in my literary aesthetics). But in a world when all of us are duped constantly and relentlessly, there is tremendous value in distinguishing between fact and propaganda.

By the way and just for fun, this blog post aligns to the following Common Core Standards:

UPDATE: Corrected a typo, ah me, but that's no guarantee you won't find another.

Thursday, January 24, 2013

Why Literature Matters: Part 1, or What Trollope Says About Assessment

I'm not sure I want to know anyone in this industry who hasn't at some time or another experienced a crisis of conscience. Humans seek meaning in our work, and unless we are psychopaths, we generally prefer doing good rather than evil. When we have a choice that doesn't much inconvenience us.

Why do our consciences rail at us during the dark nights of our souls? You know the answer. You've thought the answer yourself, or you've even asked me the question to elicit the answer, whether you be a friend, family member, or complete stranger sitting next to me on Southwest flight 1207 out of LAX.

If you are a parent or a teacher or a friend of a parent or of a teacher, or if you know and care about a child or if you ever think about what you read in the newspaper and magazines or consider what you see on the news, you know something about K-12 assessment and you probably have opinions about it, opinions that--please forgive the plain speaking, and this is through no fault of your own--are most likely ill-informed. Being as that there are few industries so shrouded in mystery and that excite so little enthusiasm and generate so little va va voom, so little sparkle as that of educational assessment publishing.

However, you probably understand that any conscious (let alone conscientious) person who creates tests for a living has concerns about the quality of the tests, the purposes for which they are used, and the consequences when they are put to use.

A potential salve to our consciences lies in 10 Reasons Why Assessments Make the World a Better Place (compiled by John Kleeman at Questionmark and sent to me by my colleague Frank Brockmann of CenterPoint--you see, we all three of us are soldiers in the same militia).

And so I, dutiful foot soldier that I am, armed with the twin purposes of promoting greater understanding of assessment and its implementation and consequences, and of reminding all of us why the study of literature is crucial to developing higher level thinking skills (and character!), will, with the aid of my betters, review these 10 Reasons Why Assessments Make the World a Better Place.

Today, let's tackle Reason #1: Assessments give equality of opportunity.

The idea that assessment ensures equality of opportunity is one that Victorian writer (and career civil servant) Anthony Trollope treats humorously in The Three Clerks (Chapter 3: The Internal Navigation) and seriously in his autobiography:

Italics mine.

SAT tutoring, anyone? How much does that cost these days, anyway? Who is most able to afford SAT tutoring? How does that disparity affect this playing field that the test is supposed to be leveling?

And here is what Trollope has to say about similar kinds of narrowly targeted test preparation:

Italics mine, yet again.

What Trollope says about himself is that he would certainly have performed poorly on any such public examination, but if he had been rejected based on his poor performance, the government would have just as certainly lost "a valuable public servant." That he was valuable is incontestable; every biographical (and autobiographical) account I've read indicates that Trollope applied himself to his career as devotedly, industriously, and with as much competence as he did to his writing. (You may not know that Trollope wrote a gazillion books, and that he rose every morning at 5 dark-thirty to write for two hours before he began his work day at his day job.)

Which ultimately means that Reason #1 doesn't really reassure me that I am making the world a better place, one test question at a time.

(In addition, a highly inconvenient truth of this industry is that the soundness of the test cannot always be taken for granted, and an unsound test, a test that is constructed in absence of best practices, should not be relied upon to produce data that is itself reliable. I predict this truth will apply to all ten reasons.)

Thursday, January 17, 2013

Whale in a Bathtub

Does anyone remember Helen Palmer's A Fish Out of Water, one of the classics from my childhood? The plot is simple, but compelling: A boy brings home a fish, overfeeds it, and the fish gets bigger and bigger and bigger until it is immense. I won't spoil the ending for you.

The need for growth management is one of the main points of what I've heard and read so far in the Coursera business class. Every owner of a successful business should carefully consider growth management. I'm thinking about small businesses now, like mine, but we've previously discussed problems associated with a failure to plan for and manage growth in larger companies. Quality is what seems to go out the window first when growth exceeds capacity.

This is not exclusive to educational publishing, or even publishing in general: it's a universal principle, as previously applied to dentistry.

Now I'd like to apply it to orthodontics. What up with the teeth? I have teen-agers, one of whom has recently been freed from braces, the other of whom is still wired up. They are twins, as you know, so why is one wired and the other free? The orthodontist's failure to manage the growth of his practice.

Like Dr. Bad Dentist, Dr. Too Successful Orthodontist came highly recommended. By three different people, one being my current dentist and another being an orthodontist in another state who went to school with Dr. Too Successful. During our consultation, he was friendly, explained everything clearly, answered all our questions, and seemed highly attentive.

Once we had signed on for a course of treatment for both my daughters, our experience turned southward. Dr. Too Successful's practice is booming. This means for a negative client experience, characterized by:

  • a loud, crowded waiting room
  • waits of up to and exceeding an hour for scheduled appointments
  • difficulty in making appointments that will accommodate one's own schedule
  • impossibility in rescheduling appointments
  • technicians who are hurried and under stress, which doesn't really bring out the best in anybody
  • necessity of having to see Dr. Too Successful's junior partner orthodontist
  • a breakdown in communication between the client, technician, and doctor, resulting in a six month extension of treatment
The latter being what happened with us. That is, we were concerned about how my daughter's teeth were slanting, I mentioned it to the technician, she said that the junior partner doctor was aware of it, and I assumed he knew what he was doing. Three months later, when treatment was due to conclude, the junior partner doctor said that whoops, some girls look really pretty with teeth that slant outwards, it makes their lips look full, some people really like that. . .but if we wanted, we could have 4 teeth extracted and start all over. That was six months ago, my daughter is still in braces, and poor thing, she may be wearing them until she is thirty.

Regardless, the main point is that Dr. Too Successful has completely lost control of his business. Soon after my conversation with the junior partner doctor, I called Dr. Too Successful to let him know that I'd experienced an erosion of trust. He said he appreciated my call and we talked about what happened. I explained that I didn't fault his office for making a mistake, but what bothered me was that the mistake was preventable, being as it was a direct result of either the technician not telling the junior partner doctor about my concerns, or of the junior partner doctor simply not paying attention.

The call concluded with Dr. Too Successful assuring me that he was dedicated to regaining my trust. And so he had seemed to be, for the next few appointments, but then his attention was captured by other clients and more pressing demands, and our experience as clients is once again suffering.

Would I ever recommend Dr. Too Successful to anyone? No, on the contrary, I would tell everyone to flee as fast as their feet will carry them. I don't like saying this, because Dr. Too Successful is likable and seems committed to doing good work. And yet, by neglecting to control the growth of his practice and by forgetting to consider his clients' experience, Dr. Too Successful caused my daughter unnecessary pain and me unnecessary inconvenience. And he has had a negative effect on my business, in that I've had to take additional time away from my work to keep taking my daughter to orthodontist appointments, appointments for which I always have to wait at least 20 minutes, and often have to wait an hour. 

Here are the questions I ask myself:
  • What is my typical client's experience?
  • Is there any negative aspect to my client's experience of Inkspot?
  • How can I improve my client's experience?

One of the little mottoes of the business course is that one has to love the client more than one loves the product. It's worth thinking about.

UPDATE: Lest anyone think I am casting stones from a glass house, I'd like to add this bit of irony. Last week, I had to call and reschedule an appointment at the last minute because my daughters had a school obligation arise suddenly that could not be missed. I was told the next available appointment was toward the end of February. The day and time offered conflicted with cello lessons, and I was offered an appointment in mid-March. By this time, feeling frustrated with having to find a time that fit my daughter's schedule, my schedule, and the straitjacketed schedule of the orthodontist, I said that I really needed the office to accommodate me and be more flexible, and here I mentioned that my daughter's treatment had only been extended to this point as a result of mistakes made in their office. I was given an appointment for yesterday.

We went to the office, and even though everyone was courteous, it was strained, formal courtesy. Something was amiss, something more than what had passed between the receptionist and me on the phone. When the doctor examined my daughter's teeth, he spoke exclusively to her, as if I weren't there.

At the end of the appointment, I opened my planner to make our next appointment and saw that WE HAD ARRIVED AN HOUR LATE. That's right. In my mind, our appointment was at 2:30, because that was the time of the original appointment, but we should have been there at 1:30 and NO ONE HAD SAID A WORD. Though their body language and tone said plenty.

I was so astounded that I didn't know what to do, but immediately upon arriving home, I called the office to offer abject apologies. I apologized about ten different ways, and I sure hope that my apologies made their way to the technicians and the doctor.

And yet? You understand that the doctor has never made that kind of apology to me for their mistake. I was an hour late for an appointment, but their mistake means that my daughter will remain in braces for what looks like a year after the original estimated end of treatment. But I guess some people find it much more difficult to apologize than others.

Wednesday, January 16, 2013


I found this interesting, about how the popularity of free online education may lead to its being free no longer, and institutions may be begin to charge students. (More here.) Which sort of defeats the purpose, doesn't it.

I'm taking a few classes through Coursera. One is a business class, as mentioned previously, which interested me because although I've had my own business for lo, these many years, I have no formal training in the management thereof. I did serve a term as a program manager at Great Big Huge Test Publishing Company, which job I liked to say consisted primarily of apologizing to clients and then begging co-workers to do the jobs for which they received a bi-weekly check. Just kidding. Sort of. I also managed the deadlines and budgets. Any successes I had in that job should be blamed on my program coordinator, who had a magical gift for inspiring people to get things done. I did learn quite a bit.

There are tens of thousands of students enrolled in the class, students of all ages (high school entrepreneurs on up), from all countries (Bulgaria, Jamaica, Chile, Italy, keep going), and of a variety of occupations. It's fascinating to read about so many different types of small businesses (wineries, vintage furniture shops, financial planning, online information management, ESL institutes, lingerie stores, you name it). I find I'm just as interested in reading the student bios as I am in the required reading for the course.

The online format is fantastic: students watch videos of lectures, reading downloaded text, and discuss questions in forums. The execution of the forum is also fantastic: the sound is good, the content is solid, the site user-friendly and easy to navigate. My only bit of constructive feedback so far is that the quizzes--from an assessment standpoint--bear about as much relation to genuine assessment as my dog does to a rhinoceros. Still and all, so far it is a worthwhile and inspiring endeavor.

Would I pay S100 to take the class? Probably not, unless I'd heard about it from a reliable source. I'm not saying it's not worth the money, it's just that I'm not sure I would have thought about trying it, and I'm not sure I was motivated enough to take that step.

Tuesday, January 15, 2013

Worry Plate

Title courtesy of Julie Allinson, founder of Eyebobs, purveyor of fashionable eyewear, from an interview provided as part of the content for this online business course I'm taking.

What's on the worry plate? Let's just say that mistakes were made. To err may be human, but nobody likes to be reminded just how human one's service provider is.

For a dozen years, I went to a dentist who had been recommended to me. He was known for being the reassuring, kindly dentist who went easy on anxious patients. His teeth gleamed, he wore a huge gold watch and always drove a newer model of European luxury car, he was hale and hearty and volubly conservative in his politics (he talked a lot about politics while performing his work; while I didn't share his views, I was loathe to engage him in debate, partly because who wants to argue with someone wielding sharp metal instruments near one's face, and partly because it was impossible to talk while holding my jaws open), and he spent a great deal of the winter skiing or talking about skiing. (Summer was dedicated to tennis.) Already in his sixties when I became his patient, the dentist died several years ago, after I had moved away.

Two years ago, I learned that this dentist had been a bad dentist. Expensive and unpleasant consequences followed this discovery. With the help of my current dentist, I tried to submit a claim for Dr. Bad Dentist to pay for the rework, but then I learned that Dr. Bad Dentist had departed this earthly realm, and that his malpractice insurance ceased covering claims a year after his departure.

So I asked my current dentist about how a consumer can tell that a dentist is good. Dentistry is highly specialized; what average consumer would be able to distinguish shoddy dental work from excellent dental work? How could I have possibly known that the cause of some of those trips to Dr. Bad Dentist was actually Dr. Bad Dentist? We talked about getting a second opinion, but what if the dentist offering the second opinion was Dr. Worse Dentist? Or what if the first dentist was Dr. Good Dentist and the second opinion was offered by Dr. Terrible No Good Dentist? There's really no way to know. My current dentist just shook his head and said you have to find someone you can trust. And I said again to him, Well, and how do I know I can trust someone?

I was thinking about how sleek and prosperous and jovial and reassuring Dr. Bad Dentist was, never failing in his good cheer, and yet--he did some damage is what I'm saying. 

My current dentist showed me some X-rays of what a filling should look like. He explained why it should look that way, and told me the principle underlying the practice. It all made sense to me. Then he showed me an X-ray of a filling of Dr. Bad Dentist's doing. He showed me the flaws and explained why they were flaws. This show-and-tell went a long way with me.

The story of Dr. Bad Dentist is just an illustration. I'm thinking about mistakes I've made and mistakes my subcontractors have made--ones that I caught before I delivered them and ones that I did not.

The mistakes are one thing--but one has to work with people whom one trusts to do their best work, to know what that best should be (and some people truly don't know and truly are unable to distinguish the good from the bad, but that is a different problem), and to fix mistakes when they make them.

From the consumer angle: ask questions. Ask a lot of them. Keep asking until you understand enough to be able to tell if the words make sense. More on this later.