Report  on  the  survey  on
student  support  services
in  the  LVU  of  FernUniversität

Georg Ströhlein²

FernUniversität in Hagen, LG DVT, D-58084 Hagen

georg.stroehlein@fernuni-hagen.de

6^th May 04

Abstract

In the framework of the international Socrates Minerva project “student support services in e-learning” a survey was carried out on students’ ratings of the importance of specific student support services and their satisfaction with the realisation of them in the “Learning-space Virtual University” (LVU), i.e. the e-learning system of the FernUniversität.  A set of 38 items has been established that is supposed to cover the most important needs of students in classroom-type as well as distance higher education.

1         Introduction

At first, it shall be briefly described what is meant by the term “student support services” (SSS).  Generally speaking, SSS comprise all services offered to students by an educational institution that are not directly related to provision and (distance) teaching of courses but are meant to support students in reaching the learning objectives.  Of course, this means SSS is a very broad subject comprising, e.g.,  academic, accommodation, administrative, advocacy, affairs, careers, counselling, disability, employment, facilities, financial, health, information, language, library, personal development, psychological, recreational, technological, transportation, tutoring, and welfare services, that obviously cannot be fully covered in a short internet questionnaire.

As FeU is on the way to establish an internet-based counterpart of the traditional paper-and-pencil distance university, only a small fraction of all possible services is already realised.  And some services that could nowadays be realised technologically may not be offered due to current laws.  Nevertheless, in a tedious process the questionnaire described in the next chapter has been compiled.

2         Description of the questionnaire

The questionnaire is organised in three main sections: in the first one the students’ ratings of their satisfaction with the mentioned items is asked for, in the second one the students’ ratings of the importance of the mentioned items is to be provided.  In both sections the items are ordered according to the steps typically occurring during a study.  The third main section is concerned with some socio-demographic data of the students.  As the questionnaire was presented nearly only to German students, its language was chosen to be German, too.  The students were notified by an email from the ZIFF.  In this email, the URL of the questionnaire together with a very brief introduction to the Socrates project was provided.  The central unit for ICT of the FeU sent the email to approximately 5.700 students who had been in contact to the LVU system at least once.

The following list shows all the items translated to English.

2.1       Pre-study phase

The items at the beginning of the first two sections are related to the pre-study phase during which an interested student tries to find out which courses to take.  Therefore, all items are related to the provision of publicly available information on the web; they are:

F01/02 WWW-info on course content
F03/04 WWW-info on position of course in syllabus
F05/06 WWW-info on the core target group of a course
F07/08 WWW-info on required knowledge
F09/10 WWW-info on students' required computer and ISP equipments
F11/12 WWW-info on learning objectives
F13/14 WWW-info on required weekly working hours (on average)
F15/16 WWW-info on voluntary or mandatory on-campus events
F17/18 WWW-info on individual advice on choice of courses
F19/20 WWW-info on costs of courses (only FeU, not ISP)

2.2       Enrolment phase

The second item group is related to the enrolment and/or course booking phase; the items are:
F21/22 valid delivery of personal data via WWW
F23/24 internet-based valid delivery of official documents (references etc.)
F25/26 valid confirmation of enrolment, personal course list etc. via internet
F27/28 receipt of user names, passwords, authorities etc. via WWW

2.3       Study phase

The third item group comprises SSS offered during the study phase:
F29/30 wide range of content of on-line study materials
F31/32 WWW-offer to download material for off-line studying
F33/34 internet-based submission of exercises
F35/36 internet-based correction of exercises
F37/38 detailedness of internet-based correction of exercises
F39/40 internet-based group work with other students
F41/42 internet-based valid confirmation of evidence of academic achievements
F43/44 internet-based contact to tutor/mentor
F45/46 internet-based contact to other students

2.4       Library services

As FeU is the only project partner offering library services, a special item group has been added:
F47/48 internet-based enquiries
F49/50 internet-based orderings
F51/52 internet-based delivery of enquiry results (e.g., via email as PDF/TIFF-file, download from a               web page etc.)
F53/54 reliable statements on delivery date of ordered work
F55/56 wide range and topicality of the part of the library store that can be enquired and ordered via              WWW

2.5       Final study phase

The fifth item group is concerned with the final phase of a course or study:
F57/58 internet-based valid confirmation of approved study work
F59/60 internet-based info on how and where to get print-outs of official documents (references etc.),                         if necessary
F61/62 WWW-info on present state of (inter-)national recognition of degree

2.6       Help systems

The sixth item group is related to the realisation of different help systems:
F63/64 internet-based help system on content-related issues
F65/66 internet-based help system on technical issues
F67/68 internet-based help system on administrative issues

2.7       Non-electronic SSS

The seventh item group addresses non-electronic student support services:
F70/71 conveniently readable print-outs of non-interactive study material
F72/73 personal group work with other students (in study centres, e.g.)
F74/75 telephone contact to tutor/mentor
F76/77 personal contact to tutor/mentor (in study centres, e.g.)

2.8       Overall level of satisfaction

In the first section on students’ satisfaction a question asking for their overall level of satisfaction has been added in order to provide some means for analysing the response characteristics:

F69 overall level of satisfaction with SSS in LVU

This single question had to be answered on a discrete, eleven-point, numerically labelled scale ranging from “-5” to “+5”.

2.9       Socio-demographic data

In the last section of the questionnaire, the students were asked to provide some personal data:
F00 usage of LVU (active, only password requested)
F78 sex (female, male),
F79 student status (full-time, part-time, guest/other),
F80 subject (electrical and information engineering, information science, humanities, mathematics,
       jurisprudence, business science, other),
F81 number of semesters in current subject (2 digits),
F82 year of birth (2 digits),
F83 state of residence (WWW-TLD-code),
F84 ZIP-code (5 digits).

2.10  Scales and layout

It has been decided to conduct an anonymous survey, and because its subject is e-learning, the WWW  seems to be the adequate medium for publishing and answering the questionnaire.  To allow for printing out the questionnaire page and filling in the answers on paper is has been decided to use a simple HTML page for publishing the questionnaire, though this opens the door to fraud by scripting techniques.  The answers were submitted using a server-based script which sent them as emails to a hidden account (which could be detected by reading the HTML source code).

The questions concerned with importance and satisfaction had to be answered using a discrete, five-point, illustrative scale with the marks “− − “, “−“, “0”, “+”, “+ +”.  The following figure shows a screenshot taken from a web browser.

Figure 2‑1:   Screenshot taken from the third item group related to the importance of             course-accompanying SSS

3         Results of the survey

3.1       Response characteristics

The questionnaire was filled in 200 times over a period of 1 week.  The first answer was submitted on Thursday, 22nd Jan. ’04 16:52 CET.  The following diagrams show different aggregations of the incoming email traffic.

Figure 3‑1: Daily aggregated response rate

Figure 3‑1 shows a rapidly decreasing daily response rate during the first week after the students were notified.  Approximately 2/3 of all answers (130/200 = 0.65) were submitted within the first 24 hours after notification.  The shape of the decrease depicted in Figure 3‑1 would have usually given rise to a reminder email after the first week.  But unfortunately, the originally hidden email account to which the answers were submitted obviously made into the address book of a computer compromised by an internet worm, so that all members of the ZIFF got many angry emails from people who had received infected emails with us as faked senders.  Therefore, the survey had to be stopped after exactly 1 week.  Additionally, the page was found to lead the list of results when the search engine Google was used to look for “student support services” and “questionnaire”, so that answers by other than FeU students could no longer be excluded.  This effect can be avoided if a questionnaire page is placed one directory level deeper in the tree, because Google is known to look only in the first sub-directory level of a top-level domain (TLD) automatically: ‘www.fernuni-hagen.de/ZIFF/befrag/some_quest.htm’ is safe of being automatically indexed by Google, whereas ‘www.fernuni-hagen.de/ZIFF/some_quest.htm’ is not.

It was then tried to answer the question whether certain patterns can be identified in the distribution of studying hours concluded from the distribution of incoming emails.  The following figure depicts this distribution in form of a stacked bar chart.

Figure 3‑2: Hourly aggregated response rate, stacked daily contributions

The distribution shown in Figure 3‑2 clearly reveals that most answers were submitted even in the first few hours after sending out the notification email: 23 = 11.5% in 1 hour, 30 = 15% in 2 hours, 44 = 22% in 3 hours, 59 = 29.5% in 4 hours, and 78 = 39% in 5 hours.  So nearly 40% of all answers reached us after only 3% of the total answering period of exactly 1 week.  A second effect which can be seen in Figure 3‑2 is the surprisingly high number of emails reaching us during usual working hours.  In fact, we even got some “out-of-office” reply emails

3.2       Characteristics of the sample

The most important characteristics of the sample with respect to a probable generalisation of the results is the distribution of subjects.  The realisations in the present sample and the total student population at FeU are shown in the next two figures.

Figure 3‑3:   Distribution of subjects in the              Figure 3‑4:   Distribution of subjects
                     FeU-student population                                             in the sample

The big differences between the distributions are most probably caused by the fact that the proportion of courses per subject which are included in the LVU system varies considerably.

....

3.3       Results concerning students’ ratings of importance

There are obviously several ways to present rankings of the items.  The most intuitive approach, which additionally does not need any assumption on type of statistical scale of the data, is to use the number of students who assigned a specific rating to the importance.  The next two figures thus show the items which got the best ten ratings and worst ten ratings.

Figure 3‑5: Ranking of items according to the number of students who recommended them most

Figure 3‑6: Ranking of items according to mean level of importance

...

3.4       Results concerning students’ ratings of satisfaction

Like for the ‘importance’ ratings, there are several ways to try to analyse the students’ ratings of their level of satisfaction with the different items.  Being interested in an answer to the question which are the items which most probably should be improved on, the most straightforward method is to rank the items corresponding to the number of dissatisfied or even most dissatisfied students.  The more or less reasonable assumption leading to this kind of item ranking is that many studies on services controlling indicate that a service user will try to change the service provider if the level of satisfaction is very low.  The next figure shows this item ranking as a bar chart.  If two or more items correspond to the same number of students, the items are ordered according to the number of all dissatisfied students or the mean level of dissatisfaction, if necessary.  This principle is used in all following ranking charts.

For each item the number of most dissatisfied students is visualised as a bar and the 95% confidence interval, which is derived from regarding the numbers as quantiles of the distributions, is depicted as vertical line.

Figure 3‑7: Ranking of items according to the number of most dissatisfied students

From Figure 3‑7 no statistically meaningful item ranking can be deduced because the 95% confidence intervals (Fleiss) of all adjacent item pairs overlap largely; only the groups comprising the first few and last few items may be found to differ.  Interestingly, there is no item at all that was not assigned a ‘most dissatisfied’ rating.

So the question arises how to construct a more reliable item ranking.  The next obvious step is to broaden the basis of the ranking by including the number of dissatisfied students; the corresponding ranking is shown in the next figure.

Figure 3‑8: Ranking of items according to the number of all dissatisfied students

As expected, the confidence intervals have become smaller now, but the situation has only changed marginally: there is no adjacent pair of items that differs significantly.  Though it is possible to identify three item clusters, F14-F40-F32, F42-F77-F46 and F26-F22-F54 being their upper boundary, centre and lower boundary, respectively, the assignment at the cluster boundaries is still somewhat arbitrarily.

The first three items in both rankings are identical, on the lower ranks more or less dramatic changes occur.  But the small changes as well as the overlapping confidence intervals between adjacent bars in both figures indicate that a statistically valid discrimination of the items is very difficult and thus a change by some ranks may not be significant.

Nevertheless, the first cluster with the items corresponding to the largest numbers of dissatisfied students deserves closer inspection and is shown in the next figure as horizontal bar chart.  Each bar corresponds to a different number of students.

Figure 3‑9: Ranking of items according to number of all dissatisfied students

We do not build a third ranking by including the number of neither satisfied nor dissatisfied students because the rating ‘0’ may merely reflect an ‘I do not know’ attitude and not an opinion on the level of satisfaction.  Instead, the mean level of dissatisfaction is tried for that purpose.

Figure 3‑10: Ranking of items according to the mean level of dissatisfaction

Figure 3‑11: Ranking of items according to the mean level of dissatisfaction

Figure 3‑12:

Figure 3‑13: Ranking of items with low mean satisfaction

Figure 3‑14: Ranking of items with high mean satisfaction

3.5       Results of modelling the satisfaction data

Supposed a ranking of items concerning dissatisfaction was known, would it then be a good idea for the institution to address the items in exactly that order? Perhaps not, because even if efforts succeed and the item dissatisfaction is removed, the impact on overall satisfaction is still unknown.  But it is widely accepted that the overall satisfaction is the important variable when customers decided on whether to leave a services provider or not.  Thus, a model has to be build to control for the influence of a single item on the overall level of satisfaction. 

Such a relationship is typically established by performing a multiple regression technique.  But the scales of the data in the present questionnaire are not definitely known to have interval qualities, so the mathematical calculations have to be applied with great care. 

As a first step is the analysis, the two-dimensional histograms of the joint distribution of all satisfaction items (5 answer levels) with the overall satisfaction (11 answer levels) were calculated and visualised as 2d-bubble diagrams to show how the data of the 200 students are distributed on the 5x11=55 grid points.  The area of a bubble corresponds to the number of students who chose the combination of items satisfaction and overall satisfaction at which the bubble centre is located.  The next figures illustrate the different kinds of relationship between item satisfaction and overall satisfaction found in the data.

Figure 3‑15

Figure 3‑16

Figure 3‑17:

3.5.1        

Figure 3‑18

Figure 3‑19

Figure 3‑20

Table 3‑1