FAQs (Frequently Asked Questions)
(Click on the triangle in front of a question to see the answer.)
1. How are we going to protect the confidentiality of faculty members? Who gets to use this data? Where is it stored?
Faculty members’ confidentially will actually be better protected than it is right now. Externally, the IDEA Center has very strict protocols that essentially say that they will send results only to a designated official of the university. In our case this will be the SRI Coordinator. No one else (at this university or any other) will be given the results. Going further, they store the data in off-line computers so there is no threat of online hacking.
At KU, our recommendations call for a local agreement that would guarantee your right to privacy. Under our recommendations, only the evaluated faculty member and peer and department chairs will receive a course evaluation report. No copies are made or kept by anyone else. All the SRI Coordinator does is to facilitate the administration of these evaluations.
2. Isn’t IDEA too long and too complicated for our students?
That is not what the students told us in the focused discussion groups, and it is not what we saw when volunteer faculty administered the instrument in their classes. We expect that the IDEA Instrument will take about 14.5 minutes to administer, and this is approximately 3 minutes longer than students take now.
The concern that the instrument was too complicated was raised by faculty members during the discussion groups. Interestingly enough, however, it was not raised by students. More to the point, the students overwhelming preferred the IDEA Instrument: 93.3% of the students favored the IDEA Instrument over the one currently being used at KU.
3. How are the additional questions to be dealt with in the promotion and tenure process?
Any additional questions added at the university level (i.e. all classes) will be approved by a local agreement and then become part of the instrument. These would be submitted as part of an evaluated faculty member’s materials.
If an individual faculty member wishes to add additional questions, then it is up to that faculty member to decide if he or she wants to include them in his or her materials. That is, unique additional questions will be used only if the faculty member wants them to be used.
4. Will the comments or open-ended questions we currently ask be asked with the IDEA Instrument?
If KU adopts the IDEA Instrument, one of the first things that will have to be done is to decide whether or not to add any additional questions. During the focus groups, faculty members expressed a strong preference for keeping these open-ended questions.
5. What is the equivalent to Question 20?
There is no exact equivalent. The closest one gets is Question 41, “Overall I rate this instructor as an excellent teacher.” The pilot program compared Question 41 and Question 20. and in our sample the mean rating of KU SRI Question #20 was 4.48. and the mean rating of IDEA Diagnostic Form Question #41 was 4.43, revealing no statistically-significant difference.
An advantage of the IDEA Instrument is that it provides a way to get beyond relying on a singular question when reviewing the students’ evaluation of his or her teaching. However, in the short-term we will have to think about acceptable standards for both instruments since some faculty will have results from both the current SRI and the IDEA Instrument in their packets. It works in our favor then that we have a good sense that IDEA provides some data that are comparable to our current instrument.
6. You seem to be suggesting that SRIs from small classes shouldn’t be used, but you still recommend the data be reviewed by the various committees?
The rationale goes back to the CBA. Article 12 requires each class to be evaluated, and we don’t have any leeway here.
What we are recommending is really nothing more than common sense. Even we who are not statisticians intuitively know that in small classes a few students’ responses can radically change the overall scores. Since we cannot put aside these results, we suggest being very cautious in relying on them.
It is worth mentioning that this problem is not tied to IDEA, but it is something we informally deal with right now without any reliability or validity studies to guide us.
7. What percentage of classes are you talking about being “very cautious” about using SRI results?
We cannot be sure. A lot depends on how the General Education model shakes out and if a local agreement on teaching in large classes is eventually reached. If we look at the data from last fall, we would anticipate around 20-30% of classes falling into this category.
8. Couldn’t we just write our own SRI or amend the one we are using?
Well we did write the current one in the mid 1970s. Records show that it took about 5 years as they sorted through different versions. It is possible, but if we went down this road, it would likely be similar to General Education reform.
More importantly, if we took this option we would not have meaningful reliability or validity studies. Contractual issues surrounding privacy of the SRIs would make any of these studies all but impossible to complete in any reasonable amount of time.
Turning to an outside tool that we know is reliable and valid and has been administered over 3,000,000 times in hundreds of schools (most of which look a lot like ours) seems like a better route to take.
9. Some courses have to be evaluated several times a semester? Are we really supposed to spend this much time on evaluations?
Results from the pilot program indicate that the current KU SRI takes approximately 11.7 minutes to administer, while the IDEA Instrument takes on average 14.5 minutes to administer. We are looking at adding about 3 minutes to our evaluation time. If you have to evaluate a course two or three times a semester, then you are looking at spending an additional 6 to 9 minutes of additional time on evaluations.
Is the IDEA Instrument worth three extra minutes of class time? We think so.
10. With the current budget problems, do we really want to spend this much money on a new SRI?
Resources are obviously tight right now. We believe it speaks a lot about this administration that they are willing to invest additional resources to improve the quality of instruction when they are dealing with tough budget cuts. They had a convenient opportunity to back away from their commitment to fund a switch to the IDEA Instrument, and they didn’t take it.
Ultimately, we are looking at spending around $12-15K on IDEA, and this is a lot cheaper than some alternatives that run $40-50K. Is this worth it?
Yes, it is because we really aren’t talking just about a new student evaluation form. What we are doing is engaging in a process to make us better instructors. We would be joining other universities who don’t simply look at SRI as a means of identifying poor professors, but see it as opportunity to improve the quality of instruction. $14K is not a lot to pay to support a paradigm shift at KU.
11. What if there isn’t money for IDEA down the road?
Contractually, the administration and the union have to agree on the SRI. If in the future the administration feels it cannot fund IDEA, it would have to work with the union to identify a different instrument. The bottom line is that the administration cannot simply walk away from IDEA without our buy-in.
12. Article 12 of the CBA requires our institution to develop “an instrument.” Your recommendations would allow faculty members to be evaluated by more than one instrument. Aren’t your recommendations prohibited by the CBA?
The clear intent of the CBA is to require the university to use a common instrument to evaluate all faculty courses. That is exactly what we are proposing. Everyone eligible in Spring 2011 would be evaluated with the IDEA Instrument.
Without grandfathering, faculty it is impossible to avoid the scenario where some faculty members are going to have the results from more than one instrument in their materials. We have been given explicit instructions from the APSCUF Contract Department that grandfathering is not an option. Therefore, if KU is ever going to change instruments (to IDEA or something else), it is going to have to work its way through this issue.
13. The instrument dumps more work on faculty members and department secretaries. Can’t our time be better spent?
The opposite will be true. By removing the deans from the process and having the SRI Coordinator facilitate the process from beginning to the end, there will be considerably less confusion and work for everyone.
Using free Premark software, the Faculty Information Forms will be largely completed before reaching the evaluated faculty member. PET chairs and secretaries will no longer have to count forms and complete course header sheets. Instead they will receive a packet for each of the evaluated courses. All that is left to do is for the evaluated faculty member to mark any additional objectives and the department to administer the evaluations.
14. We just changed the evaluation period. Do we really want to change it again to simply accommodate an outside vendor’s evaluation instrument?
No, we don’t want to change the evaluation period. The committee was unanimous on this point. However, we were equally convinced of the need to make sure that the forms are processed and returned to the evaluated faculty in time for the contractually mandated promotion deadlines.
So the switch to weeks nine through eleven is really just a safeguard to make sure that faculty members have their material back in time. If it turns out that the SRI Coordinator can manage this faster, then there is no reason we can’t move the date back to something closer to what it is now.
15. The pilot program did not meet minimal social scientific standards. Why should we value the data it generated?
Clearly, we would have liked to have had greater participation from the faculty both in the focus groups and also in the administration of surveys. However, there were serious limitations (e.g. contractually, only tenured faculty who were not standing for promotion or in their fifth year of review could offer their classes) and time constraints given everyone’s teaching obligations.
What we did was to try our best to make sure everyone was informed of the process and knew they were welcome to participate. Ultimately, 702 students and 68 faculty members participated in the pilot program, and that isn’t too bad.
16. What if the evaluations are lost in the mailing process?
The completed evaluations will be mailed via a major carrier (e.g. FedEx, DHL, or UPS) and will be tracked. If the unexpected happens—and we have a Tom Hanks’ Cast Away moment where he and his friend Wilson are playing with their new friend “KU SRI”—the evaluated faculty members are protected. Article 12 of the CBA requires that the administration process the evaluations and if they fail to do so, then the affected faculty member will be excused that semester.
17. I have no idea what a raw, adjusted, converted, weighted score is…and why are you are recommending the use of the adjusted converted averages?
There are really only two things one needs to know.
(1) Scores are weighted. If you mark a learning objective as “not important,” then it is given a weight of zero and it drops out. “Important” is given a weight of one, and “essential” is double weighted. You control what objectives count and by how much.
(2) Scores are then adjusted and converted for comparison purposes. That is, adjusted scores take into account factors beyond your control (e.g., size of class and student motivation), and converted scores account for the fact that some objectives score higher than others on average (e.g., obtaining factual knowledge scores better than critical thinking).
We recommend using the converted weighted score because it does two things. It measures how much progress the students believed they made on the course’s most important objectives, and it adjusts for factors outside the control of faculty. Large classes with unmotivated students will no longer weigh down the scores of good instructors.
18. One of the first things any student learns in a statistical methods class is that you don’t average ordinal level data. Isn’t that exactly what IDEA does?
Yes and no. The data generated by the IDEA Instrument is ordinal level data, and the reports automatically report averages and percentages. As a general rule, we agree that “Statistical Methods 101” suggests that you don’t average such data for very good reasons. That is, if all you are doing is ranking preferences and you don’t know the exact distance between the preferences, then averages could be very misleading.
IDEA essentially argues that this is messy data. They have produced a series of studies that show that over time averaging scores actually makes sense. We can forward these technical reports if you like.
Also, it is worth noting that this is basically what almost all of us do now. We report our average Question 20 score.
Finally, if you find these arguments unconvincing, then one can always use the “raw” class evaluation data. With this data, it would be relatively easy to report median scores and be in compliance with the strictest methodological guidelines.
