ALBUQUERQUE, N.M. — Professors long have known that using students to evaluate courses and teachers can produce less-than-useful data that can be misused by administrators who may give it too much weight in deciding whether a course or instructor is on the right track.
The person in charge of curriculum at the University of New Mexico – himself a professor – was gratified recently to learn of new research that shows evaluations are often misused statistically. He agreed that while traditional student evaluations can be useful, their value is limited.
“We’ve recognized that for some time,” said Greg Heileman, associate provost for curriculum at UNM.
The research in question is a draft paper by two scholars at the University of California at Berkeley, which was highlighted in the Chronicle of Higher Education in September.
According to the paper, student course evaluations rarely shed much light on the quality of teaching or the value of a specific class. Yet, at the same time, they are often misused statistically.
“We’re confusing consumer satisfaction with product value,” said Philip Stark, a professor of statistics and one of the authors, in an interview with the Chronicle. Stark teamed up with Richard Freishtat, a senior consultant at Berkeley’s Center for Teaching and Learning, not only to critique student evaluations of courses but also to submit an alternative method of analyzing and enhancing teaching practices.
Just about everyone recognizes that there is some value to the evaluations, especially when patterns emerge that can lead to constructive change. The main problem is that university administrators often apply them disproportionately in the promotion and tenure processes.
The main criticism is with the math behind the evaluations. Results can be skewed by such variables as response rates; ethnic, gender or age bias; and the common practice of averaging and comparing scores.
“Averages of numerical student ratings have an air of objectivity simply because they are numerical,” the authors write.
“Such a practice,” said the Chronicle, “presumes that a five on a seven-point scale means the same thing to different students, or that a rating of a three somehow balances with a seven to mean the same thing as two fives.
“For teaching evaluations, there is no reason any of those things should be true. … Such averages and comparisons make no sense, as a matter of statistics.”
The authors suggest that instead of averaging scores, distribution and response rates of the evaluations should be reported. That way, a cluster of scores would be more informative than a general average.
Heileman, meanwhile, said UNM has already begun moving away from traditional hard-copy student evaluations toward an online model. Like the authors, he was also skeptical of some of the traditional evaluation questions.
“Is the student in the position to judge the value or worth of a course while he or she is still enrolled?” he asked, rhetorically. “I don’t think so.”
UNM’s College of Arts and Sciences has developed a broader and ultimately more honest way of evaluating individual professors – a five- to 10-page teaching portfolio. In addition to student evaluations, the portfolio includes a personal statement of teaching philosophy, a record of past teaching, documentation of course development, peer evaluation, a description of how a teacher’s scholarship and teaching interact, and other activities important to a clear representation of the professor’s teaching work.
In the past, UNM used a system developed by a Kansas-based nonprofit, IDEA. According to its website, IDEA’s “flagship service, the Student Ratings of Instruction System, focuses on student learning of specific objectives. With IDEA, you receive both summative and formative feedback. The data can be used for individual faculty as well as program and institutional development.”
While that may sound good, the UNM faculty, as a whole, doesn’t like it, Heileman said. Professors and instructors find it too complicated and not all that useful or online friendly.
UNM is currently considering other online evaluation systems, including:
⋄ EvaluationKIT, which asks administrators, “Do you want a simple and robust course evaluation system that plugs directly into your Learning Management System? EvaluationKIT has turn-key solutions that make your course evaluation project setup a breeze, and allows seamless access for students and instructors from directly within your LMS.”
⋄ eXplorance, which advertises “Fully integrated & automated software that adapts to your organization’s needs.”
⋄ CollegeNET, a Portland, Oregon-based, company that boasts “web-based on-demand technologies to colleges, universities, and non-profits. … We currently serve over 1,300 higher education and non-profit institutions world-wide.”
A decision will be made soon.