A Web Based Module Evaluation System Interim Innovation Project

a web based module evaluation system interim innovation project report to the learning & teaching committee nadine foster (dept of physi

A Web Based Module Evaluation System
Interim Innovation Project report to the Learning & Teaching Committee
Nadine Foster (Dept of Physiotherapy Studies), Nikki Rochford (Dept of
Physiotherapy Studies), Barry Smalley (School of Chemistry & Physics),
Stephen Bostock (Staff Development & Training).
Aim of Project
To develop a web-based system to facilitate the rapid analysis of
module evaluations. The module evaluation system will be based on one
currently in use by the Physics Department.
Background and Purpose
The Department of Physiotherapy Studies has demonstrated a keen
determination to improve the already high standard of quality
assurance mechanisms used within the department. Whilst the department
feels its present mechanism of module evaluation is effective, it’s
reliance on paper questionnaires and the resultant time required in
the analysis and production of useful data, renders this process
rather intensive in terms of staff time and of questionable
cost-effectiveness. The relevance of these issues in a quality
assurance programme is recognised and the aim of this project was to
improve the efficiency and cost-effectiveness of the module evaluation
process.
We further proposed to develop this into a portable system using the
Physiotherapy Undergraduate course as a pilot for a more general
system. This evaluation system will also facilitate the use of
information technology by students and will respond to the needs for
impartial collection of student views without unnecessary use of staff
time.
The Department of Physiotherapy Studies further recognised that the
paper module evaluation forms were not fully student-centred. It was
felt that they did not consistently investigate students’ attitudes
towards their own responsibility for the learning process. It was
therefore intended that the modular evaluation form developed for use
on the web would investigate the extent to which the student actively
engages in the learning process.
We also recognised the need to be able to compare the students’
learning experiences across modules. It was expected, therefore, that
the web-based module evaluation system would facilitate comparison in
the student experience across modules. The development of this
web-based module evaluation system, within the Physiotherapy
Department would act as a pilot to evaluate the transferability of the
system to other departments within the University. This will fit well
with the aims of the Quality Assurance initiatives to enable
comparison across departments and Faculties.
It was further expected that this project would contribute to the
embedding of IT within the BSc (Hons) Physiotherapy programme, as well
as the promotion of the transferable skills of life-long and active
learning, via student reflection.
Objectives
*
To develop a standard module evaluation form for the BSc
Physiotherapy Studies (Hons) programme, which would facilitate
comparability across modules
*
To further develop & enhance the supporting software to analyse
and present the evaluation data producing meaningful web-based
summary data.
*
To develop a user-friendly tool which does not require a high
degree of computer literacy to use
*
To pilot the developed evaluation questionnaire forms within the
Physiotherapy programme
*
To investigate how feasible it might be to use this web-based
system of module evaluation across the wider University.
Method
In order to meet the above objectives, the following procedure was
undertaken:
1.
Current module evaluation tools used within academic environments
were reviewed for their appropriateness and pertinence,
additionally focus groups and staff discussion were utilised to
identify the main areas considered to be important in a module
evaluation.
2.
From these a draft questionnaire was then developed.
3.
The draft questionnaire was which was reviewed by all staff and
representatives of the student body for relevance and
appropriateness and further developed on the feedback.
4.
The developed module evaluation form was then adapted to a web
based system, which was then piloted within all first semester
physiotherapy modules.
Attached to this report are examples of the form and the outputs.
Results of Pilot
1.
Comparability between web based and paper evaluations.
To ascertain if the computer results were comparable to the results
gained from a paper evaluation the student cohorts were randomly
assigned to two groups, the first undertook the computer evaluation
and the second completed the same evaluation form but on paper.
The following table gives a comparison of the results obtained from
these two sources.
Modal score Modal score
for web version for paper version
Module 1 4 4
Module 2 4 4
Module 3 4 4
Module 4 4 4
Module 5 4 3
Average number of students giving a low score (1,2) for a question,
out of about 38 students (and range of values for any one question)
for web version for paper version
Module 1 2.7 (range 0-22) 6.7 (range 0-31)
Module 2 7.1 (range 0-25) 5.5 (range 0-22)
Module 3 2.5 (range 0-20) 3.5 (range 0-18)
Module 4 8.5 (range 0-22) 9.7 (range 3-17)
Module 5 5.1 (range 0-14) 7.7 (range 1-15)
It is clear that the method of delivery did not systematically affect
the results obtained from the module evaluation.
2.
Staff Evaluation
A questionnaire was completed by all module leaders to determine their
views of the new system:
Staff preference
All 5 staff preferred the new evaluation form
2 staff preferred computer system to the paper based one
3 staff stated no preference between the two systems.
Benefits
All staff stated that the main benefit of the new system was the
amount of time saved.
The ease of access to results was also considered to be a benefit.
3.
Student Evaluation
Students also were asked to complete a questionnaire concerning their
attitudes towards the new evaluation system. The following graphs
illustrate the results of this audit.


Overall, students were happy with the new system. A larger number of
students who had completed paper evaluations stated a preference for
paper, but they had had no experience of the web based system. Those
who used the web form, on balance, preferred it. They were positive
about the design of the form. Various improvements were suggested,
many of which had already been considered. A large majority of
students felt the medium made no difference to their answers, and
those that thought it did were not consistent in the effect they
thought it would have. A majority thought it would be necessary to
book a PC lab to complete the web forms were the computer system
adopted.
Further developments
*
The standard web form is now being used for all physiotherapy
modules, Years 1, 2 and 3.
*
A programme evaluation questionnaire had additionally been
developed and was piloted at the end of May 2002; the results are
pending.
*
The web report now provides summary data for inclusion in the
course report. The usefulness of this will be determined after
summer.
*
The transferability of the system to other departments is now to
be determined.
Conclusion
It is expected that, with minor amendments, this method of module
evaluation will be embedded within the plans for the new BSc (Hons)
Physiotherapy course due to commence October 2002. This is expected to
result in a more cost-effective and efficient method of evaluation.
Added benefits for Physiotherapy Studies include the increasing use of
IT by students and the improvement of information quality gathered as
part of module evaluation. The system design will explore long-term
issues such as ensuring portability. It is intended that the
transferability of this tool to other departments will have been
explored and recommendations made for campus-wide availability.
05/10/21