PILOT STUDY

The main goals of this Pilot Investigation centered on (1) identifying administrative concerns associated with adopting IDEA (e.g., amount of time required to administer IDEA Instruments and how to handle selecting course learning objectives) and considering how we might handle these concerns; (2) obtaining student opinions, including preferences, of the KU SRI and IDEA Instruments; and (3) obtaining faculty opinions, including preferences, of the KU SRI and IDEA Instruments.

In brief, the Pilot Investigation revealed that:

  • The IDEA Diagnostic Form will not take that much more class time than our current instrument, and it will provide information that will assist in improving teaching and learning practices.
  • IDEA instruments offer a question comparable to our current question #20.
  • The IDEA Diagnostic Form was clearly preferred more than the IDEA Short Form or the current KU SRI, especially by students. Indeed, students indicated that they view our current instrument as woefully inadequate in capturing their opinions regarding teacher evaluations.
  • There are several administrative concerns that must be addressed if Kutztown University moves forward with one of the IDEA instruments.

Summary of Participant Information

In total, 702 students and 68 faculty members participated in some aspect of the Pilot Investigation.

FOCUS GROUPS: There were 8 student focus groups, consisting of 57 students, and 5 faculty focus groups, consisting of 31 faculty. All four colleges, College of Liberal Arts & Sciences (CLAS), College of Business (COB), College of Education (COE), and the College of Visual and Performing Arts (CVPA), as well as undergraduate and graduate students, were represented in the focus groups.

ADMINISTRATION OF SURVEYS: A total of 37 course sections, involving 645 students, participated in administering both the KU SRI and the IDEA survey instruments to students. 20 of those sections administered the IDEA Short Form; 17 administered the IDEA Diagnostic Form. Faculty from the CLAS, COE, and CVPA participated in this phase of the Pilot Investigation. Unfortunately, despite extensive efforts to involve faculty from the COB, the Implementation Committee was unsuccessful in this effort.

Administration Time

  • KU SRI – Mean administration time: 11.7 minutes (range: 7 – 19 minutes)
  • IDEA Short Form – Mean administration time: 9.5 minutes (range: 6 – 15 minutes)
  • IDEA Diagnostic Form – Mean administration time: 14.5 minutes (range: 9 – 20 minutes)

Comparison of Question #20 to comparable IDEA Questions (#17 and #41)

Question #20 – KU SRI

All things considered, how do you rate the instructor’s performance?

(a) Excellent (b) Very Good (c) Good (d) Poor (e) Very Poor

IDEA Question (Diagnostic Form #41 and Short Form #17)

Overall, I rate this instructor an excellent teacher

(1) Definitely False (2) More False Than True (3) In Between (4) More True Than False (5) Definitely True

In sections that received the IDEA Forms (N=37), the mean rating of KU SRI Question #20 was 4.53 (SD = 0.44) and the mean rating of IDEA Question was 4.52 (SD = 0.39) revealing no statistically significant difference in ratings. Each section of a course rated the instructor twice, once with the KU SRI and once with an IDEA Instrument. Furthermore, the order in which courses received the instruments, administered on different days, was counterbalanced.

The data points in the figure below indicate the overall correspondence of the data for these KU SRI and IDEA instrument questions. Specifically, the mean ratings obtained for the KU SRI instrument and the IDEA instrument in each participating course section are plotted thereby revealing their statistically significant correlation. The trendline indicates the degree of linear relationship between the data points. (Please note: Only 32 data points, rather than all 37, are evident in the figure below because there were 5 instances where more than one course received identical scores on the two instruments, thereby resulting in overlapping points.)

These data reveal that the questions (KU SRI #20 and the IDEA question) possess similar mean values, are significantly correlated, and possess a strong linear relationship.

Student and Faculty Preferences for Evaluation Instruments

STUDENT PREFERENCES: Students overwhelmingly indicated that they preferred the IDEA Diagnostic Form. Across the seven student focus groups for which these data were recorded, 42 of the 45 students (93.3%) indicated that they preferred the IDEA instruments. Furthermore, an analysis of the comments indicated that the majority of these students preferred the IDEA Diagnostic Form.

FACULTY PREFERENCES: Unfortunately, preference data was not consistently recorded from all of the faculty focus groups. Three of the five focus groups clearly recorded preferences for the IDEA Diagnostic Form with only two of the three recording specific vote totals (six votes for IDEA – zero votes for KU SRI). Of the two focus groups who did not record preferences, one did not record any specific preference information as the focus group discussion wandered to address other issues. The preference data for the remaining focus group were ambiguous with a unanimous preference for the IDEA instrument recorded at one point whereas at another point the vote was recorded as split between the IDEA instrument and the KU SRI. Nevertheless, collapsing across all focus groups, a general preference for the IDEA Diagnostic Form emerged. This preference, however, was not as overwhelming as that observed with the student focus groups.

Please see the complete report for more details about the Pilot Study.