EvaSys was introduced as the course evaluation system at the London School of Economics and Political Science (LSE) in autumn 2009 and has been successfully used for School-wide evaluations in December and April. EvaSys allows the LSE to choose flexibly between paper and online surveys and to provide improved reporting to stakeholders.

Surveys were first introduced at LSE to evaluate teaching in the early 1990s and recently, the School had relied mainly on online surveys. “There were two main goals when introducing EvaSys at the School”, Mike Page, Head of ARD Systems and Business Processes at LSE, explains “We wanted to improve the timeliness and accuracy of survey results, and to encourage higher response rates. EvaSys is empowering us to achieve these two goals.” The EvaSys suite automates many of the steps that normally need manual interaction, making the whole process from survey creation to reporting significantly more efficient. This means staff get feedback on their teaching much faster and at the same time it enables the School to use paper based surveys on a large scale to maximise response rates.

In 2009/2010, modules were evaluated in a three week period towards the end of each term so that students have had a substantial amount of teaching at the time of the survey. In the Michaelmas term surveys were run evaluating over 400 graduate teaching assistants and 200 permanent teachers resulting in 20,000 questionnaires being processed. In the Lent term some 25,000 questionnaires were completed, surveying over 600 courses for over 500 permanent teachers across all modules.

Standarised questionnaires are used throughout the School so that results can be easily combined: one for permanent teaching staff, covering questions on the quality of course contents and its teaching, and one for graduate teaching assistants which goes into more detail to evaluate their teaching in the classroom. Both questionnaires give plenty of space for open-ended responses, which are presented automatically in the reports.

For each course to be evaluated, a coversheet that identifies course and teacher is generated based on the details imported into EvaSys from the timetabling system. The coversheets are attached to each pack of questionnaires. The surveys are then administered in class and/or lecture and returned by a student in the envelope provided to a drop box in the Student Services Centre to ensure confidentiality. LSE has contracted Electric Paper to provide remote scanning services. Once the completed surveys are received, the Electric Paper project team log into the LSE hosted EvaSys system and scan approximately 2,500 - 3,000 per day during the survey administration. At the end of each day the LSE staff runs the EvaSys automatic batch reporting function to generate and email out instant individual PDF reports for each course/teacher combination.

For teachers, the instant feedback report presents a detailed analysis, displaying summary statistics for each closed question graphically and numerically, alongside the answers to open questions. When all surveys across the School have closed, teachers receive an additional report with profile lines showing how their individual results compared to the department and School overall average. The profile line comparisons also show their combined scores across all the courses they teach.

Heads of Departments receive copies of the individual reports, including only the quantitative results, i.e. without open comments. They also receive the raw data so that they can conduct further analysis in any statistical package if they wish to. In addition, reports with aggregate results showing quantitative scores of all courses within a department and an aggregate for the School have been produced and sent to Heads of Departments, the Director of the Teaching and Learning Centre and Pro Director (Teaching and Learning).

Turnaround times improved dramatically with all results returned before the end of term, whereas previously some staff had had to wait weeks for their results. Staff much appreciated the more timely feedback while the course is still fresh in their minds and also commented positively on the clear visual presentation of the results. Mike Page explains “The swift turnaround of clear and accurate results was a key requirement for the new system and EvaSys more than met expectations in this area.”

The flexibility of EvaSys in terms of delivery mode (paper or online) is another significant advantage for the LSE. As evaluation results are used in promotion decisions, high response rates are a priority in order to reduce the potential for bias and inaccurate data. Paper based surveys are also preferred because past experience has shown that they tend to yield more feedback in terms of comments.  Response rates in recent years for online surveys were typically less than 50%. The paper survey in Michaelmas term received a response rate of 72%, while in the Lent term the response rate for classes and seminars was 75%.

For online surveys, it is also possible to integrate EvaSys with the existing student portal at LSE, Moodle, so that students can respond to online surveys directly from their Moodle accounts. Paper and online surveys are administered in a unified system, so that changes or a mixture of survey modes are accommodated without extra work. This flexibility of the system was also pivotal in the selection process. “EvaSys was selected after an extensive tendering exercise because it offered the most flexibility yet dedicated system for our needs”, says Mike Page, “It’s a clear winner in our eyes.”

Next year, the administration of the surveys will be organised more flexibly to ensure that teachers can conduct their surveys outside of this main survey period at a time that is most appropriate for their particular teaching pattern, e.g. where one teacher only teaches the first half of a course before another teacher takes over.

LSE have now successfully completed two survey administrations in their first year. Issues that inevitably arise when dealing with such large volumes of paper for the first time, and also with different combinations of teachers/courses/classes/ lectures were resolved swiftly with the help of the Electric Paper team. Overall, the biggest benefits from implementing the new system for the LSE are the turnaround speed and the ease of use of the system, especially with regard to reporting. The attractive, easy to interpret reports are another major benefit for the School as they help staff to make the most out of the feedback they receive from students.

Feedback from Permanent Teachers and Graduate Teaching Assistants

“[...] The [questionnaire] forms, the speed at which they were returned, and the ease of understanding the box markings and comments were EXCELLENT and I really, really appreciated them. So thanks!”
“Thank you very much. I appreciate the speed of processing/feedback -now that the course is still fresh- and the visual displays.”

“Having just received my course evaluation results, I would like to express how helpful the surveys are. The new procedure, in which we hand out the surveys in the class, is a far more reliable way of getting student responses, providing (hopefully) a more accurate reflection of class teaching and areas where we need to improve. The new format for presenting the results is also much more effective.”

“Many thanks for making these useful changes to the feedback system and for the swift turn-around of the results - much appreciated!”

“I also wanted to thank and congratulate you for your extremely efficient organisation and handling of the survey process - I thought it was absolutely exceptional, and I was very impressed throughout the whole process.”