Student questionnaires have been an inescapable part of academic life for quite some time, probably longer in the USA than here. My first (temporary) teaching position was at the University of Michigan in 1973, and this is where I met questionnaires for the first time.
It was a learning experience for me. I read the comments on the first questionnaire and was delighted; this student praised me highly, said I was the best teacher he had ever had. So I turned to the second, and was brought down to earth by the mirror image; I was the worst teacher he had ever had, and he explained why in some detail. I was lucky; this experience inoculated me against the disease of taking questionnaire comments too seriously!
My present employer has just changed the questionnaire system. They have done an unbelievably bad job; everyone I tell the story to gets angry about a different part of it.
The background is that the department ran the questionnaires for many years. The questions had evolved over time to give a reasonably good assessment of what the students thought about all aspects of the course. A new question was added after some debate to test how new technology (especially course web pages) was being used.
Then, halfway through the academic year, the College decided that new questionnaires would be introduced, which would be identical in all subjects. Departments were given almost no say in the matter. If we wished to continue our existing questionnaires, they should be in parallel with the College ones. Correctly, we decided not to overburden the students with bureaucratic nonsense, and withdrew from the field.
The questionnaire consists of seven statements; students are asked to rate each on a five-point scale from “Definitely agree” to “Definitely disagree”. On the back, there is space for comments, segregated into several fields. So here are some of the things wrong with the system. I have put first the one that troubles me most; but you may have a different view.
1. One of the statements is
Through studying this module I understand what I need to do to be successful in the assessment
None of the statements says anything about understanding the material or learning anything. So I infer that the College’s official view of my job as a teacher is to ensure that the students to do well in their exams. If that is the case, this is not an institution I want to be part of! In any case, rather than soliciting opinions on this, they could wait a couple of months and look at exam results to obtain more objective data.
2. In the old system, questionnaires were done in week 6 of the semester, and the results processed quickly. If there were serious problems with a module, the management would become aware of this in time to do something about it. In the new system, the questionnaires are filled in in weeks 9-11, and because they are read by computer the processing takes much longer. It was only by taking a stand that we were able to get the first few results returned by the end of week 11. This is far too late for the feedback to have any effect.
3. The questionnaire design is bad. In both our old questionnaires and the National Student Survey, “good” answers are on the right; on the new ones, they are on the left. I know for a fact that some students simply got it wrong.
4. This is compounded by the way the data is processed. In the old system, after the answers had been tallied, the questionnaires went back to the lecturer for comment. There is some chance of catching problems like the above. If the numerical answers are all extremely bad, but the comments on the back are very positive, you might guess that this had happened. But now the comments are detached from the tickboxes and the different fields separated out by the computer that reads the forms, so this kind of deduction is no longer possible.
5. The most absurd thing about the data processing is this. With the old form, the summary statistic was the median answer to each question; the only reasonable thing to do with ordinal data. In the new system, the software that reads the boxes calculates the mean and standard deviation. Then it plots the seven means on a diagram, and runs a curve through the points(!). When I asked about this, I was told, “Not everybody understands the median.” Oh dear.
I believe this happened because they bought in the entire system, fancy paper for the questionnaires, software to process them, etc. The cost of all this would probably have paid for another teacher, which might have had more effect on the student experience.