January 29, 2007
We’re in an era when many professors fear that student evaluations — either the formal kind sponsored by colleges or the informal kind found on places like RateMyProfessors.com — may play too large a role in whether they earn tenure or raises or, in the case of adjuncts, whether they are hired back.
A new study by three economists at Ohio State University may add to those fears. Previous studies have found that students are more likely to give good reviews to instructors who are easy graders or who are good looking. The Ohio State study — in many ways larger and more ambitious than previous ones — found a strong correlation between grades in a course and reviews of professors, such that it is clear that students are rewarding those who reward them.
That finding alone, however, may not negate the value of student evaluations. One explanation could be that good students are earning good grades, and crediting their good professors for their learning. The Ohio State study, however, provides evidence for the more cynical/realistic interpretation — namely that professors who are easy (and aren’t necessarily the best teachers) earn good ratings. The way the Ohio State team did this was to look at grades in subsequent classes that would have relied on the learning in the class in which the students’ evaluations were studied. Their finding: no correlation between professor evaluations and the learning that is actually taking place.
In another finding of concern, the study found evidence that students — controlling for other factors — tend to give lesser evaluations to instructors who are women or who were born outside the United States. And they found this despite not finding any correlation between instructor identity and the level of learning that took place.
While there may be ways to improve the reliability of student evaluations, the authors write, “we believe that any student evaluations are best used in conjunction with peer reviews of teaching.”
The study was just released by the National Bureau of Economic Research. (An abstract is available here, where the paper may also be downloaded for $5.) The authors are Bruce A. Weinberg, Belton M. Fleisher and Masanori Hashimoto.
In their study, the authors evaluated data from 50,000 enrollments in 400 offerings over a period of years of principles of microeconomics, principles of macroeconomics, and intermediate microeconomics.
While the study is generally critical of the accuracy of student evaluations, it suggests — when looking at both grading and learning for correlation — that there are other possibilities for explaining why students’ evaluations don’t correlate with actual learning.
One explanation, for example, is that students don’t themselves have a good sense of how much they are learning. The authors stress that there are many ways — such as adjusting for student bias for easy graders or bias against certain groups of instructors — to continue to use student evaluations as one tool for measuring professors’ performance. But they write that, used alone and unadjusted, they appear highly questionable.
jbs, at 6:55 am EST on January 29, 2007
Without reading the abstract or the paper, I don’t know that it’s wise to take a position other than the one in the last paragraph—you need to triangulate measures of teaching with multiple opportunities for peer observation and review, a self-assessment, and perhaps the information gained from alumni. And who would disagree with that? But let’s not be too hasty to agree with the study—it’s likely to be based on the information from one department—i.e., one discipline—at one institution, one of the largest in the country. And the information is from students of a discipline notoriously difficult conceptually. And we know nothing about the type of evaluation tool or tools being evaluated. Are they homegrown? Are they commercial? What kinds of questions are asked? How can the authors say that they looked at “grades in subsequent classes,” when student evaluations are almost always anonymous? How can you track changes in student learning when you don’t know who your learners are? Having said that, I admit the methodology looks interesting, and the study looks worth trying to replicate.
Dean Rich, Bethel University, at 7:00 am EST on January 29, 2007
Assessment of student learning, of faculty, and of institutional or course performance involves much more than taking a “width of the grin” survey. There is a role for such surveys inside a comprehensive strategy, but any school or department using this kind of measure by itself deserves what it gets.
Ward Deutschman, at 7:20 am EST on January 29, 2007
George, at 7:25 am EST on January 29, 2007
In the past I have suggested, though it is not as easy to impliment as I would hope, student tracking through disciplines. A student who takes an algebra class from professor X and gets a B will take the next class from professor Y and get an A, perhaps. Put that information into the hopper with the other students who passes professor X’s class and the other students who entered professor Y’s class, and you start to get a picture of how well students are doing, rather than how well students think they are doing.
Andrew Purvis, at 7:45 am EST on January 29, 2007
The use of student evaluations is the most controversial and divisive issue in academic life.
In my experience as a teacher and reenforced in courses I have audited by other faculty members, students are increasingly empowered. They are far less respectful than they used to be, they are much more demanding, and they are much less responsible for such things as attendance, promptness, exam preparation and turning in work on time.
The problem is that what students want is not identical with the training school is supposed to provide. There is certainly some overlap, but students are busy people and unless there are clear and consistent rules the class turns into a bartar system where everything is subject to negotiation. We are now negotiating special times for exams because people have scheduled vacations during finals week. In the week prior to midterms, we debate in class exactly what will be covered on the exam and how much information the students will be allowed to bring with them to the classroom. We debate how much the students will be expected to memorize. Sometimes we even debate the content of courses.
In calculus I have had to debate whether I was entitled to explain the definition of a derivative or to use it to obtain the differentiation formulas. In a college algebra class I found myself debating whether I had the right to explain such processes as completing the square or deriving the quadratic formula. Sometimes students press their point to the extent of constantly interrupting and saying “why are we doing this?” and “is this going to be on the exam?".
All of this would simply be part of the job if the use of evals in judging teaching performance hadn’t made appeasing such behavior part of the criteria for merit pay, getting hired and receiving promotion and tenure.
When I began my current job in the late 1980’s, I looked at the exams that had been kept on file for the past twenty years. I noticed that the level of what was being asked of students had gone down substantially. I find that on any number of topics that I teach, what I expect from students has diminished over the years.
What I have noticed is that the single most important criteria for improving evaluations is lowered expectations. As a rule of thumb don’t ask difficult problems on exams, don’t ask them to do problems that aren’t exactly like the ones they have seen, hand out practice exams that are virtually indistinguishable from the actual ones, spend your class time doing examples of problems that they can commit to memory for the exam and don’t spend time explaining the proof of a difficult theorem, and above all don’t be “too theoretical".
Unfortunately many of my colleagues have weaponized the student evaluations and the scores are now considered to be the definitive measure of whether one is a good teacher or not.
Over time, evaluations have gotten extremely high. But rather than recognize this as a good thing, the evaluation process uses the “departmental average” as the way of distinguishing between the performances of different professors. If the department average on a 1-5 scale for a question is 4.5 and you receive a 4.2, then your work is substandard.
Most students are reasonably honest about trying to answer the questions. But if 20% judge the teacher on whether they give in to their demands or not, they are the determinants of whether the teacher scores above or below the departmental average. The process iterates itself every quarter or semester as teachers anticipate the complaints and adjust their expectations so as to avoid the humiliation of giving in to the manipulations of students who want to take the exams at special times, miss class and come to office hours for the lecture, allow the students to bring in sheets with all the formulas on them, drop the lowest grade, drop the two lowest grades, tell them exactly which topics will not be on the exam, etc.
There has been a great deal of discussion about how we assess student performance. If we care about the education of students perhaps we should first discuss how we are assessing teaching. Peer review is not necessarily an answer. The students, as subjective as they are, tend to be more objective than colleagues whose judgment is influenced by friendship, personal likes and dislikes, and most of all by the fact that they are judging people with whom they are in direct competition.
I don’t have an answer to this problem. Teaching is for most faculty the most important part of the job. It is important to do it well. But finding criteria for judging good teaching that will actually enhance the performance of teachers rather than hinder it is not an easy task. If it was, we would have done it already.
Jonathan Cohen, Professor of mathematics at DePaul University, at 7:55 am EST on January 29, 2007
While suggestions that student evaluations be considered in a wider curricular context have some merit, a look at them in the wider institutional context may reinforce the arguments of those who feel that the educational experience is becoming more and more a shopping excursion served by an entrepreneureal retail sales staff of professors and a growing deanery of customer service agents. Whatever the issue, academic integrity and intellectual honsety seem less important than customer satisfaction.
However reassuring to the shopper, none of this is, finally, helpful to the student.
j ranelli, at 8:00 am EST on January 29, 2007
With regard to anonymous student evaluations of faculty, my own argument has never been dependent on the claim that “research shows this” or that “research shows that.” It has always been that anonymous, unsigned opinions of people — whether flattery or complaint — are worthless and that we should not just ask but require our students to behave like adults. If they have a complaint, they should write it down and send it to the teacher. It’s a matter of principle. The college doesn’t have to invite students to write anonymous opinions of teachers and to then send these opinions to the faculty’s bosses. Everyone is already free to do that if one wants! But very few people do that because a vast majority of us understand that to do such a thing is obviously unethical and immoral and cowardly. When the college puts its imprimatur on such a process, however, and also demands that teachers participate in it — by distributing the institutional forms, for example, and leaving the room — it persuades students to believe that this obvious evil is is both useful and good.College students are not children. The are adults, accountable for the actions and opinions.
Bob Schenck, at 8:45 am EST on January 29, 2007
Student evaluations taken immediately after the class are worthless.
Find the student five years after graduation and ask what was learned. And, that too must be suspect until the questions are refined by experience.
Self evaluation by the instructor and supervisors is the best measure. The quality of student has a great deal to do with what can be achieved. One test fits all is off target.
Good citizens must be at the top of the list. Not much public support for the job done to date. Sorry record when it took 78 years to free the slaves, 112 years for woman to vote, and we continue to solve political problems by war.
William Sumner Scott, J.D.
Judicial Equality Foundation, Inc.
William Sumner Scott, J.D., at 9:16 am EST on January 29, 2007
With regard to anonymous student evaluations of faculty, my own argument has never been dependent on the claim that “research shows this” or that “research shows that.” It has always been that anonymous, unsigned opinions of people — whether flattery or complaint — are worthless and that we should not just ask but require our students to behave like adults. If they have a complaint, they should write it down and send it to the teacher. It’s a matter of principle. The college doesn’t have to invite students to write anonymous opinions of teachers and to then send these opinions to the faculty’s bosses. Everyone is already free to do that if one wants! But very few people do that because a vast majority of us understand that to do such a thing is obviously unethical and immoral and cowardly. When the college puts its imprimatur on such a process, however, and also demands that teachers participate in it — by distributing the institutional forms, for example, and leaving the room — it persuades students to believe that this obvious evil is both useful and good. College students are not children. They are adults, accountable for their actions and opinions.
Bob Schenck, at 9:16 am EST on January 29, 2007
Please if students were required to sign, their evaluations the faculty would surly retaliate against the students who gave bad grades. The because of the way faculty uses tenure to cover incompetence the students need autonomy to protect them selves. The best way would be to for colleges hire good market research firms that know how to get reliable answerers to difficult questions – their skills get tested in the market place every day.
Stephen Wells, at 9:17 am EST on January 29, 2007
Do faculty retaliate against students who complain? Do students retaliate against faculty who give them bad grades? Hey, it doesn’t matter. Either allegation can be written down and, if necessary, investigated properly. At some point the accuser must face the accused. Anything else makes the accusation chickens—-.
Bob Schenck, at 10:10 am EST on January 29, 2007
I don’t have the $5 required to see the full article...however, my guess is that with 50,000 students in the sample many of the correlations that the authors found are spurious (appear only due to the large power that a sample of 50,000 has). I recommend interested readers see the free (from most libraries) and balanced article on the validity and reliability of student evaluations written by d’Apollonia and Abrami (1997). Navigating student ratings of instruction. American Psychologist 52(11), 1198-1208.
Jeremy, at 10:10 am EST on January 29, 2007
For the time being, the paper can be downloaded for free.
Prof. Guy, at 10:10 am EST on January 29, 2007
In our department, peer evaluations are considered along with student evaluations. However, faculty are unwilling to be honest in their evaluations. They give strongly positive evaluations, then deny them during the closed tenure-review meetings. Thus a faculty member cannot take a strong review at face value, but must wonder what was not said and what will be said behind their back. However, if you do not receive both strong peer evaluations and strong student evaluations, you are considered insufficiently competent. The assumption that any achievement must be the result of “grade inflation” in the reviews denies faculty any satisfaction from doing a job well, and induces paranoia about what will be said about them by their peers. There is no way to address comments made in secrecy.
A premise of teaching is that people learn only with feedback. If teachers are denied honest feedback about their performance, how can they improve? Peers will not provide honest feedback, then torpedo their colleagues behind closed doors.
Carlos, at 10:35 am EST on January 29, 2007
Students can get very different grade outcomes over the same course materials by taking different professors. Grades are the most important student incentives. So, there will be classes where students can get better grades, and students will definitely prefer them by-and-large. To expect otherwise is foolish.
The problem is we have no way to distinguish between professors who have dumbed-down the material and professors who are good teachers. In both cases, one will get grades in the requisite statistical configurations, and good student evaluations.
Jeff Younger, at 10:35 am EST on January 29, 2007
What kind of evaluation did you give the professor who neglected to teach you to proofread what you write?
Professors don’t see these evaluations until after grades are in. How can they retaliate? Besides, I already know which of my struggling students take it out on me because “the work is too hard,” “I don’t like this,” or “I have more important things to do than the work in this course.” I don’t need a signature.
Judith, at 11:30 am EST on January 29, 2007
After a dozen years of full-time teaching and another two dozen as a chief academic officer who continued to teach at least one course each year, I am very disappointed with the discussion and the continuing need to validate instructional paranoia. First, all instruments of evaluation of faculty must be designed by the faculty to ensure their ability to provide helpful feedback for instructional improvement. We did and its works very well in the eyes of the faculty and greatly expanded their use.Last, In all my above years, I rarely see a student evaluation that is critical of gifted, inspiring, and effective faculty. It is much more common to see references to difficult material, harsh grading, and excessive work in the same evaluations that the students express love and awe of the faculty member. I am convinced students provide very real and useful feedback.
David, CAO, at 11:30 am EST on January 29, 2007
Why not have the student’s transcript consist of anonymous critical essays about the students written by their professors and the teaching evaluations consist of a single letter grade awarded by the students? In their current form, neither the anonymous comments or letter grades seem to represent anything of substance or descriptive or predictive value.
Petet, at 11:53 am EST on January 29, 2007
Without change in our opinion that standard achievement tests and student opinion are both worthless, if they are to be used:
Have the student (i) take a test before the class begins to measure incoming achievement level and (ii) write a paper of what is expected from the course.
Then the tests during the course and final are compared with a paper by the student to explain what was accomplished at the end of the course.
Try to get the process as subjective as possible. The studies we have conducted proved that the bottom of the class was there when they arrived and did little to move to the top. Their evaluations should be thrown out for sure.
To avoid retaliation, the papers in and out can be identified by student number only. Try to make the best of a bad situation — higher education is suspect at present.
Higher Education is more of a social standing achievement measure (who gets into bones) rather than a productive citizen effort.
William Sumner Scott, J.D.
Judicial Equality Foundation, Inc.
William Sumner Scott, J.D., at 11:53 am EST on January 29, 2007
This study is not surprising. Years ago, a truce was developed between students and faculty. If faculty would not challenge students and ensure that most of them get decent grades, students wouldn’t bother them too much, freeing up their time for research, consulting, Monday through Thursday work schedules, and no classes after 2pm. Meanwhile, if students would not bother faculty or hold them accountable for learning, faculty would reward them with indifference or good grades. Occasionally, an engaged professor is paired up with a motivated student, but these instructors are often seen as “change agents,” and are rare in higher education. An instructor who embraces new learning methods is rare.
This missing part is accountability. If instructors worked in environments where they held each other accountable, real learning would occur and external intervention from entities like the Spellings Commission would be unnecessary. But holding colleagues accountable for learning is unheard of in academic departments. These people are quickly labeled as opponents of academic freedom and tenure and are vulnerable to departmental bullies. The path of least resistance is more certain and easier. If students held each other and instructors accountable for learning, they would worry more about what they learn than a grade, an artificial construct that says little to nothing about learning. A perfect storm, indeed...
PS, at 1:20 pm EST on January 29, 2007
i have been retired for 13 years after teaching general physics and physical science at Auburn University for 34 years. I got a lot of bad evaluations (along with some really good ones) from students who found me too demanding. I refused to lower my standards on the grounds that I probably knew better what they would need to know and the problem solving skills they should develop that the typical freshman or sophomore did. i even told some classes that I wasn’t there to be their friend, but to help them learn. I wasn’t very popular, but at least some of the students expressed appreciation later on.
Charlotte Ward, Dr. at Auburn U., emerita, at 1:20 pm EST on January 29, 2007
Instructors who want good evals should not discount the personality factor. I am a much more difficult instructor than many of my peers—teaching harder material with harder exams and higher standards when grading. Yet I consistently get higher evaluations than many of my easier peers. The key is engaging the students directly. Talk to them like people, get to know them, be up beat and don’t bore them to death.
Scott Hendrix, Professor at Pellissippi State, at 1:21 pm EST on January 29, 2007
Maybe grades, not student evaluations, are the problem. Educator and psychologist Alfie Kohn argues against extrinsic motivators like grades. He thinks education should rediscover students’ intrinsic motivation to learn: namely, their natural curiosity and desire for mastery. He offers other ways of assessing this last. See, for example, his book _Punished by Rewards_: The Trouble with A’s, Praise, Gold Stars, Incentives and Other Bribes_.
Randall Spinks, at 1:21 pm EST on January 29, 2007
Some 20 years ago, when I told a good friend who was at a PhD granting institution that I had won a college teaching award at my school, there was a shocked silence on the other end of the phone line — initially no congratulations, warmth, happiness for me. Cautiously he asked me how I felt about getting the award. I was a little surprised but said I was very pleased. His apologetic response for his previous lack of enthusiaim for me was that, in his department, getting such an award was, in his words, “the kiss of death” — there would be an immediate assumption by other faculty coleagues that you were spending too much time on teaching and not enough time on research! Tenure would almost certainly be denied.
I recently served a few years as department chair. The faculty who got the worst teaching evaluations and most student complaints, always justified their performance by claiming exactly what this new study claims, that they are the ones upholding the standards. By extension of this idea — not a logical equivalence but a belief reinforced by this study — the faculty who DO get good evaluations or teaching awards, must be easy graders, with low standards. From what I could observe as chair (reading ALL thousands of written evaluation comments of every faculty in the department as part of annual evaluations), this is blatantly wrong and totally unfair. With very few exceptions, the faculty getting the good evaluations are the ones doing the better teaching jobs — and not the easier graders, or by lowering standards, etc.
Manny, at 1:21 pm EST on January 29, 2007
End-of-course evaluations by students have been controversial for decades. This approach to evaluating courses and instructors has well-documented advantages and disadvantages, most of which can be favorably addressed by adhering to best practices in survey design and administration. Yet there is another approach to improving the validity and utility of student evaluations — doing more with them after they are collected. For example, evaluation results for given courses, departments, or instructors can be “normed” over time, thereby revealing trends where they occur and differences where they are important. For example, if required courses consistently garner lower scores (or more critical write-in comments) than elective courses (even in the same department), this factor can be taken into account when a faculty member, department chair, or rank & tenure committee is assessing student evaluations for a particular course or instructor. Similar “norms” can be calculated for types of instruction (e.g., lectures vs. labs vs. tutorials), or for student levels (freshman vs. seniors vs. graduate students), or for fields of study (where, for example, GPA’s usually are lower in the sciences than in the humanities). While this type of comparison might seem tedious, it should be fairly easy to provide when standardized, machine-readable scoring methods are used. Unfortunately, too many (perhaps most) departments, colleges, and rank and tenure committee meetings fail to ask for such readily calculated comparative data. Instead, student course evaluations often are “read” in an ad hoc manner that undercuts comparison and therefore devalues much of the worth the questions themselves might have yielded. Let’s hear from some colleges and universities that creatively and effective use best practices in post-survey analysis of student evaluations.
Richard Yanikoski, President at Assocation of Catholic Colleges and Universities, at 1:21 pm EST on January 29, 2007
When I think of students evaluating their instructional experience, I think of my attempt to evaluate, say, a particular surgeon. Someone might ask me, for example, “Do you like that doctor?” And I might answer, “Sure, he’s nice. He answers my phone calls. He explains what I don’t understand. He made me feel comfortable in the hospital.” What happens when I find out years later that he removed my kidney and not my gall bladder? My point is, students know when they are receiving poor service. They can tell if an instructor cares. But they often cannot tell if they are receiving a poorly written, standardized ("canned") curriculum, superficial instruction, wrong information; likewise, unless they are told, they might not be able to tell if the instructor is being “tough” for a reason, if the instructor is presenting fact or opinion, or if the instructor is presenting bias as if it were truth. So student evaluations might be good tools in assessing a students’ understanding of the course materials (which is better done mid-semester than at the end of the semester), but I am not convinced they depict an instructor’s value or effectiveness in teaching.
I once worked at a career school that decided any instructor who did not meet the benchmark on student evaluations would be fired. I am not exaggerating, and you can imagine the outrage among the faculty who often had to struggle with poor equipment, administrative incompetence, and set curricula while trying to teach effectively and earn high grades from the students. Even worse, I was the one who was supposed to enforce these policies without even questioning the questions on the survey (which were generated by the corporate office, by the way). Um.....no. Not why I went into education. And I doubt teachers who value teaching went into the field knowing they would be hired or fired based on these standards.
kgotthardt, at 1:21 pm EST on January 29, 2007
David suffers from an opinion poll of one and the rest of us suffer from not knowing where he conduction his poll.
We have evaluated numerous achievement methods and read reports of many others. They all believe that a student can form an evaluation only after some distance from the classroom. Rarely, if ever, do they believe their instructor is gifted — only the very top of the class come close to recognition of quality instruction. The middle to bottom are usually indifferent.
William Sumner Scott, J.D.
William Sumner Scott, J.D., at 1:21 pm EST on January 29, 2007
I don’t teach anymore, but I’m encouraged to see the thoughtful responses to this story. Prof. Cohen: My 11th grade son has DePaul on his list of maybes, I hope he goes there and has you as an instructor...
viejita del oeste, at 1:21 pm EST on January 29, 2007
One of the comments implied that “customer service” surveys are somehow beneath the academic intent of our institutions. My frustration is that more schools do NOT take the customer service focus more seriously. You CAN bring both capabilities together — academics and customer service — to the benefit of both the students and the institution. We need to keep our students in school, to complete their maturation as contributing adults, and to bring a promise of life-long learning. Only if we find value as adult students will we continue the process.
The old style of “sit down and take this” is not relevant to the newer generations, and certainly not appealing to this life-long learner.
Faculty that are fair, respectful towards their students, engaging, challenging, honst and collaborative will score higher with me and probably affect more students’ lives.
Steve, at 1:51 pm EST on January 29, 2007
It is interesting that if or when administrators view student evaluations of faculty in context of an honest assessment of student learning, they’ll discover that students are learning very little. That’s a frightening prospect in light of outcome based education performance standards imposed by legislators.
Despite whatever faculty do to encourage active learning, students are passive consumers. Education has become a transactional rather than a transformative experience: Studens regard their degree as a consumer durable of which they only have to spend time and money and not spend intellectual involvement.
I quit reading my evaluations after I was promoted to full professor; I couldn’t stand anxiety brought about by reading comments from students who don’t have any involvement in class other than to sometimes occupy a desk and pay tuition. Now I just throw them in a drawer.
Curt Stofferahn, at 1:55 pm EST on January 29, 2007
About 1980 the AAUP published two studies on student evaluations. I’ll try to summarize the results which strike me a still useful. The first study interviewed graduates ten years after leaving school. The respondants generally rated the faculty they gave low evaluations to as the better faculty. Reason: the low evaluations were given to demanding instructors. Now that they were in thw workplace the knowledge the demanding teachers imparted was valuable. The faculty given high evaluations generally were likeable because they were less demanding. They taught the students less.The second study first identified those qualities making for good teaching: engaging and non hostile teaching style, excellent knowledge of material, approachable and available to students. Poor teachers were confusing, hostile or non approachable and so on. Then the faculty idintified the good teachers and bad teachers for the same subject and studied the grades the students earned on standardized tests for the subject. The expectation was the students of the good teachers would do better on the tests. The performance of the students was inversy correlated to faculty rating. Then the students were intervied and what was found was the typical student in the spell bindingly good teachers’ class said to himself he really understood the material and he’d better put in more time on the books for the poor teacher’s class. In short, it was the amount of study time on the books that correlated directly with the acquisition of knowledge and not the teacher’s ability.Student’ evaluations are a “beauty” contest not much more meaningful than the Miss America contest.
Frank Cannonito, at 2:10 pm EST on January 29, 2007
The real question about student evaluations and how they are utilized is not whether they accurately measure teaching performance but whether over time they have substantially alterred how we teach and if so is it for the better.
The contention of my earlier post is that it has had over time the effect of diminishing what we accomplish, both in terms of what the students have learned of the content of our courses and in terms of the work habits that students are developing in our classes.
Contrary to the anonymous CAO David’s assumption that he can learn all he needs from the student comments, a great deal of what he refers to as “faculty paranoia” is simply an honestly expressed frustration over the compromises that faculty have made to make sure that their evaluation numbers are pleasing to their chair or dean. That is something that you can’t learn from the student comments or numbers because the students don’t know what choices their teachers are making.
My experience with this debate is that the most vocal people are those who feel they have the largest stake in it either because they get unusually high evaluations or unusually low ones. This is unfortunate because it is at the extremes that the evaluations are probably the most accurate. The faculty member who gets great numbers and comments that indicate the course was very tough is probably doing a great job. Similarly, the faculty member who gets continual complaints that the teacher is incomprehensible, unavailable and returns papers a month after collecting them, is probably doing a lousy job.
The point is that most teachers fall in the middle somewhere and these are people for whom the daily decision making of what to lecture on, which problems to assign, how difficult to make the exams, how much advanced information should be given out on exams, and which special favors to grant (do you ignore absences, excessive tardiness, failure to show for midterms) are filtered through considerations of how it will affect their student evaluations. The reason teachers consider how the students will react is because they are correct that it will play an important part in how they are judged. Over time it is very difficult to continue to make decisions that you know from experience will hurt your evaluations.
The purpose of using student evaluations to measure teaching is to pressure teachers to do what it takes to maximize their scores. After all, why would we use them if we weren’t prepared to say that a person who scores 4.5 is doing a better job than a person who scores 4.4?
This means that after doing the good things such as preparing better, making sure papers are returned promptly, making sure you are scrupulously fair, you start working on ways to improve the scores by doing things the students want but you may not think are such good ideas: assign only odd numbered problems that have answers in the back of the book, give generous partial credit, make sure that as many answers as possible are positive integers between 1 and 5, assign problems 1-9 odd instead of 17,35,46,50 and 53 and so on.
I have been a department chair and served on personnel committees and hiring committees where evaluating teaching takes place. And based on my experience you can not tell from either the scores or the written comments, how much an individual instructor is gaming the system. And even in the case of someone who is brazen about what he is up to, there is absolutely no sanction for their behavior. We have a system of evaluation that is out of control and administrators, like David, CAO who commented above, are living in dreamland if they think otherwise.
Jonathan Cohen, Professor of mathematics at DePaul University, at 4:36 pm EST on January 29, 2007
A persistent problem in traditional university course grading, especially where instructors are convinced of their own objectivity, is that exams are set and graded by the course instructor. This amalgamation of roles does not seem to be well understood in the discussion of student evaluations. I take it that it is plausible to assume that performance in an examination is a function (among other things) of student abilities, teacher abilities and examiner/grader abilities. But if this is so, it means that in grading an examination on my own course I am implicitly also either ignoring or grading my own teaching (and my own examining/grading), putting the student at a disadvantage. There is more to be said about this, of course, and a separation of the instructor and examiner roles will not be possible for all courses. But wherever it is possible, this separation of instructor and examiner/grader roles will remove one of the personal elements which contribute to the much-discussed vicious circle of evaluation-grading dependencies.
Dafydd Gibbon, at 4:36 pm EST on January 29, 2007
PA in PA, If students are customers and education a product, at 4:36 pm EST on January 29, 2007
I appreciate many of the comments posted here. The evaluation as most of us know it is a computerized survey with little room for real reflection or comment by the student. It does indeed reduce the classroom experience to that of a MacDonalds meal, and as such, places the student in the position of consumer and the instructor in the role of french-fryer or cashier.
After reading Bill Readings’ discussion of the evaluation as a place to ask students to think seriously about their own learning process, I dumped the campus’ evaluation form last year, and used an essay format, which I gave students a big chunk of time to do. This action threw the registrar into a frenzy, because they didn’t know how to process these texts, and the deparment still hasn’t given these evals back to me, because they literally don’t know how to use them.
I’m a tenured prof, and so can afford to do these kinds of “experiments” but I think that deparments, grad student orgs, and unions need to strategize around this issue, and decide or at least debate what measures these bodies will and will not accept as valid accounts of how well teachers are teaching.
Stephanie Hammer, UC Riverside, at 4:36 pm EST on January 29, 2007
This study shows that faculty (and others) in higher education need to have the right to bargain collectively regarding terms and conditions of our jobs.
When administrators have the power to impose the results of student evaluations (often using instruments that have never been evaluated to determine the reliability and validity) and faculty have no ability to impact how they are used, we have a power imbalance that only harms us. If we have the right to negotiate, we at least have a chance to make it right.
I often wonder why administrators are never evaluated by students, faculty, staff or anyone, for that matter. What are they afraid of???
Wade Hannon, Member at AFT Local 4660, at 4:36 pm EST on January 29, 2007
I have friends who have taught at West Point, private Catholic K-12, and unionized taxpayer-subsidized colleges.
At West Point — it is the Army way, or the highway. Whine about high standards to mommy’s Senator, and a free, one-way ticket out is offered.
At Catholic schools — argue with the bishop about pedalogical approach, get a one-way ticket back to the Public School Monopoly.
Unionized public colleges? Once tenure kicks in, who the frack cares? It takes a B-level felony to be fired. That attitude shows — big time.
B.D., at 7:01 pm EST on January 29, 2007
When are people like B.D. going to stop throwing about generalization that insult and degrade tens of thousands of serious educators. I have taken the time to read the narrative portions of evaluations, and students who have taken the time to include them have given me useful information that I have incorporated in later terms.
What I have learned is that I can push students because I tell them up front that I am going to push them, that I am going to make them write that little bit more because it is largely through the processes of writing and revision that they will improve. The texts, my lectures, and our in-class work will help out with the tools, but it is a demanding writing schedule that, if the students stick with it, will do the most.
My students groan on the first day. They thank me after the final. No, I am not the greatest educator on the planet, or even close. Yes, I, like many of my colleagues in both public and private institutions, care about helping the students learn (forget the myth that we can pound things into them).
Evaluations often reflect this, but students who feel as if they are hitting the wall without understanding why they are being asked to face it are likely to get a little surly. If we want students to appreciate the academic standards we demand, we need to explain it and help the students see that the very things that cost extra hours in homework will pay real dividends over the years. Change that and you’ll change much of this disconnect with regard to evaluations.
Andrew Purvis, at 4:10 am EST on January 30, 2007
I have examined many comments from students who also rated my teaching on a scale of 1 to 5 with ‘5′ reflective of ‘Superior’ teaching ability.a. “This Professor is horrible. The students have to do all the learning in this class"Rating = 0 on scale of 1 to 5b. Dr. P. is a good teacher and knows his stuff, but he requires you to put most, if not all, the effort in learning the material."Rating = 3 on a scale of 1 to 5c. This is not your typical course, You have to actually LEARN the material."Rating = 5 on a scale of 1 to 5.
These were on undergraduate courses which were required. They covered Learning, Development, Motivation and so on.
evan powell, at 7:35 am EST on January 30, 2007
I would ask one question, however. What exactly are we supposed to be teaching? Is our focus to teach content only or to prepare them for work life after college? If we define our function as broader than content (as I do), what are we teaching them about their position and power as employees by allowing them to anonymously evaluate faculty?
I am aware of more than one situation where a new employee ran into trouble for assuming he/she had evaluation (and veto) authority over a supervisor. Do we have any responsibility to teach them that they may not understand or like it, but that’s life? They will face this for the rest of their lives — when do they start learning how to effectively deal with it?
The true measure of effectiveness in teaching content AND life skills cannot be measured while students are still in school.
CM, at 10:25 am EST on January 30, 2007
Higher or k-12 education, the same problem, and the same solution: to change the philosophy that educational institutions are serving students (or students are customers, schools are student-centered, etc). Students come to schools to be trained like athletes, not served like customers. Receiving education is a training process and one must work to learn. No pain, no gain. It’s that simple!
The median and the society have lots of discussions on this issue, and few of them hit the point. But global competition is a objective comparison: our kids are getting worse and worse, at least, academically. In a college wherein my friend is teaching, their college tennis team’s athletic studentsGPA is significantly (one letter) higher, not only than other sport teams, but also than the college wide average. You know why? It is not because playing tennis may help one to have better grade, but because more than 90% of the tennis players are from other countries: developed as well as less developed!
Using students evaluation to determine faculty members’ teaching quality and hence salary or tenure/promotion is one of those policies that really hurt our education quality in our society! It will in fact hurt our students and hence our future in the long run! Unfortunately, very few people (including policy makers) understand this point!
I am an excellent instructor and I know the materials I am teaching. What is more, I really love teaching and my students! But I am forced not to teach them the best I can. Why? Because of teaching evaluation! I care about my students’ learning, and I also have to care about my evaluation as well. The outcome: I have to focus on how to raise my rate of evaluation rather than how to teach well and what (most important materials) to teach. I did experiments quite a few year ago and learned it hard way: the more time and effort I put on preparing for serious teaching, the lower rate I got! Then, I simply set a goal to do anything I can (legally) to maximize my rate of teaching evaluation and it worked for me very well. Though I am having a much better rate than before, I am not proud of it; rather, I feel guilty about what I am dong. But I have no choice. For example, one simple rule to make students learn is to set a demanding attendance policy that either forces or induces students to attend classes. I knew it! But when I adopted it, my rate dropped significantly! Then, I changed it not to require attending, my rate rose immediately. I know it is not good, and not what I want to do, but I don’t want my chair or dean call me a bad instructor. As far as I know, all of my colleagues have their own tricks to raise their rate, though none of them actually is really good for teaching effectiveness at all!
I really admire a former colleague of mine who can earn very high rate from her students and some students of hers told me that her class (of accounting)is very demanding as well and students believe they have learned a lot. But in the 5 years interval when I was there, none of her students have passed CPA exam! Note that she was a teaching excellence award winner!
However, I really appreciate the comments I received from my students. In fact, I myself learned one of my bad habit from (verbal) comments from my former students that I liked to talk to the blackboard while I was writing. Since then, I never talk while I am writing any more.
It is overdue for our society to re-valuate our practice in education and to eliminate student evaluation is probably one of the most important reforms we need to think about. I also understand that in many European countries, instructors are also evaluated by their students, but it would not affect their salary or promotion, or even be known by their boss at all!
Matthew York, at 10:41 am EST on January 30, 2007
“What I have noticed is that the single most important criteria for improving evaluations is lowered expectations.”
Lowered expectations? You seem to expect that your students will be a bunch of snotty, grade-grubbing consumerists. Your evaluations must be through the roof!
jcl, grad student, at 11:05 am EST on January 30, 2007
We must not always assume that when an instructor gets a bad student evaluation that it means the student is doing poorly.
In my undergraduate experience, I gave negative evaluations when instructors deserved them, regardless of how I did. I never got less than a B in college, so I can’t say I ever did poorly, but there were times I gave positive reviews in classes where I earned a B and some times I gave negative reviews when I earned an A.
In one class, I had an instructor who was repeatedly not in his office during scheduled office hours. Many times I would show up to ask questions about the material (it was a math class) and nobody was home. I mentioned this when I completed my student evaluation of the course, and to this day, I make no apologies for doing so.
While some students do not study and work hard, there still exists a problem when faculty dismiss negative evaluations from students who do have legitimate complaints. Too often some professors don’t bother to keep their scheduled office hours, don’t show up when students schedule meetings, refuse to answer questions, etc., and if the student then gives a negative evaluation, it is dismissed as being from a “bad student.”
Faculty members who behave this way see themselves as being superior to students and use the “bad student” excuse as a form of immunity from criticism. We need to consider, however, what would happen if one were working in the private sector and didn’t keep scheduled meetings with fellow employees. Inevitably, they would get a negative performance review (or get fired), and they wouldn’t be able to chalk it up to a “bad employer.”
jrob, at 4:01 pm EST on January 30, 2007
” .. If we want students to appreciate the academic standards we demand, we need to explain it ..”
When is Mr. Purvis going to get enough work experience (and common sense) to accept the reality of too many marginal colleges accepting too many marginal students?
http://www.opinionjournal.com/extra/?id=110009531
A graduate assistant at a Tier 1 college can try to set high standards (and be disappointed).
Try that a Tier 4 college, and you’ll be back at Starbucks, faster than you can say “but all I did was ..”
B.D., at 7:25 pm EST on January 30, 2007
Maybe this is too obvious, but student anonymity on evaluations should cause no difficulty in tracking subsequent performance. You know the roster of the class and presumably who missed the evaluation. You simply track everyone’s grades in the subsequent course(s) and average, which is really what you’d presumably do anyway. With a large enough statistical pool of students, grading by a range of professors in the subsequent course should be a reasonably objective measure of which professors’ students are best learning the things expected of them in subsequent courses. If enrollment in the initial course is blind, any anomalies in the students entering a course should smooth out over a number of sections, too.
The entering pool may be affected some by professor reputation, though, as students who manage to elbow their way into a difficult-to-get section with a famous professor (if any still teach freshman and sophomore courses) may already know how to play the academic system better than others, which probably will also mean higher subsequent grades.
Time of day of courses may affect things, too. At a community college, often the majority of daytime students are recent high school graduates, while the evening students are older persons coming to school after work. The grade profiles of these two groups can differ quite a bit.
Thane Doss, Yomiuri Culture Centers, at 5:45 am EST on February 1, 2007
I hate to say
I hate to say “Told you so"... but I did, and so did a lot of other people.
Jonathan Dresner, at 6:30 am EST on January 29, 2007