JISC, in delineating the areas of work for the two Strands, set out the following objective for Strand A:

As indicated, the sampling frame considered size of institution and type of institution (Old university Russell, Old university non-Russell, New university and College of Higher Education) (Appendix 4). Various categorisations were considered, but the sampling frame chosen seemed to cater for possible differences in research and teaching emphasis. The large, medium and small categories were chosen arbitrarily, after reviewing some lists of HEIs used by JISC and the sampling frame agreed with the JISC Scientific Adviser at the first meeting with the adviser in September 1999. Sources consulted included:

The actual process of selection was done by numbering institutions as listed in HEFCE, SHEFC, and HEFCW listings, obtained from the Web sites:

Random numbers were selected, using a random numbers tables. With each selection, the size and status of the corresponding HEI was checked using lists of Russell group members, project team knowledge of former polytechnics and HE colleges, and data from Appendix 3 to the MAU March 1999 report. The selected HEI was added to the appropriate cell in the table, working on a maximum of 3 HEIs per box. If a box already contained 3 names then a further random number was drawn. There were three cells for which no HE institution could be found to fit. Bangor was allowed as a fourth HEI in the cell for Old university, non-Russell/medium, to include representation from Wales. The sample spread was also checked against THES statistics: average RAE score (1996), library expenditure and computing expenditure.

A similar procedure was followed to populate each discipline cluster. The discipline clusters were derived from the RAE 2000 subject grouping lists, and, again, various categorisations attempted, with reference to groupings that have been used for other purposes. The source used was:

The five discipline clusters eventually chosen seemed to be inclusive. Another consideration was to obtain comparable staff and student numbers (as a proportion of the total population) for each discipline cluster.

The departments were selected by taking one discipline cluster at a time, picking an HE institution at random, and then listing and numbering the appropriate departments for the cluster under consideration, for further random selection. The selected department was then added to the list for the cluster. Adjustments were made for the mix of departments within clusters as necessary by redrawing department numbers as necessary. Ten HEIs per cluster were selected (target 6 departments) to allow for non-participation.

One difficulty was the problem of departmental division as the University Web pages for some institutions did not subdivide below Faculty, or sub-Faculty level, and descriptions included Faculty, School, Division, Department, and Centre. Care was taken to ensure a reasonable balance of types of institution for each subject cluster. At this stage, the Humanities and Arts, Pure & Applied Science and Clinical Medicine categories were satisfactory, but adjustments were made to Maths & Engineering (by adding a randomly chosen HE college to represent HE colleges), and to Pure & Applied Social Science (deleting at random 2 from 5 New Universities (over-represented); adding Russell (chosen at random); adding one non-Russell (chosen at random)). Later adjustments were made when it was found (after contacting the HEIs) that, for example, Aston: Life & Health Sciences has 3 departments (Pharmacy School the selected for subject coverage), and Plymouth: Arts & Design is a course, not a department, requiring selection at a level higher up i.e. Faculty of Arts & Education.

The original intention had been to obtain the staff and student lists from the departments and randomly select names for the discipline cluster from those lists, with the proviso that a minimum number of research subjects would be required at each site, staff and students, to allow for cross-checking of the teaching/learning use of EIS.

Data protection limitations on access to student names and/or e-mail addresses at some institutions meant that various approaches to access were arranged. Departments are, of course, given assurances that all names and e-mails will be treated confidentially and results anonymised in reports provided to JISC. The approaches to accessing names and means of contact were:

• Full list of staff and student names provided to JUSTEIS team (as originally envisaged) (route 1)
• Department makes a random selection of ‘x’ number of students, contacts the students, who may volunteer their participation, either directly to us or via the Department. There is usually no problem in obtaining names of staff (though full contact details are not always provided) (route 2)
• Department or Central Information Services sends out a blanket e-mail to all staff and students in selected departments asking them to volunteer participation, directly to a JUSTEIS contact (route 3)

The covering letter (e-mailed) provided for the purpose of mediated access was similar to the first covering e-mail (Appendix 6) sent to advise the selected HEI departments, and the appropriate LIS (Appendix 7), but also contained information about an incentive to participate (inclusion in a Prize Draw) (Appendix 9). That letter (e-mailed) was sent to the subjects selected at random from lists supplied directly to us.

There are advantages and disadvantages to each mediated approach: route (3) is obviously easier for most institutions to implement, but it is more difficult for us to control the balance of type of student and year required, as well as biasing the sample of those more likely to use electronic information services. Route (2) is more work for the departments but slightly easier for us to ensure that there will be the proper mix of students within the sample. There are slight variations on both routes depending on institutional policies.

The survey instrument (Appendix 10) comprises:

• Basic details checklist (categories used were agreed with the JUBILEE project)
• Questions (2, 3, 4, 5) dealing with a recent critical incident of information seeking. The questions explore the background to the incident, the purposes and motivations, informal sources who prompted the search or were consulted, time and access constraints, and degree of success in obtaining information required.
• Link question (6)
• Questions (7, 8, 9, 10) dealing with the critical success factors and the priorities placed on certain electronic information services as a result.

There is a checklist (to ensure that all varieties of EIS have been adequately covered by the interviewer, and also a main question checklist (to ensure all questions were answered as some interviewees answer later questions in the course of answering an early question).

The pattern of purposes and interaction in question 4 were used as prompts in the interview, but are primarily for later analysis. This is drawn from a taxonomy for higher education information services (Saracevic, 1997a, 1997b). Both the Saracevic taxonomy and the EIS checklist (derived directly from the JUSTEIS taxonomy) were used in the analysis.

The pilot study was conducted in three departments in University of Wales, Aberystwyth during the second month of the project. The aim of the pilot was principally to test the survey instrument, and the procedures involved in sampling and gaining access to students. The site was chosen for convenience, although it was realised that some of the procedures for accessing students and staff in other HEIs would not be tested fully.

The departments chosen were School of Management and Business, International Politics, and Biological Sciences. These provided a suitable spread of subject fields (in comparison to the main categories of subject areas chosen for the main phase), a range of type of staff and student (including some distance learning students), and, as all departments were on one campus, some of the logistical problems for student interviews were overcome. A room was provided by Information Services for the interviews with students (UG and taught Master’s PG). Staff and research postgraduates were interviewed by telephone.

Procedures used were those planned for the main phase:

• Contact Head of Department and Administrative Officer, giving details of the survey
• Obtain lists of students and staff, with e-mail addresses
• Take random sample of staff and students
• Contact selected subjects by e-mail and/or phone to arrange a time for interview, following up non-response where appropriate.

The distribution of the pilot sample is shown in Table 5.1.

A total of 30 interviews were arranged and 28 were conducted. Of these two were re-arranged after ‘no-shows’.

Interview transcripts were loaded into NUD*IST for qualitative data analysis. For the main phase some of the analysis was quantitative but the main aim of the pilot analysis work was to refine the coding categories to be used, and develop a suitable command file for NUD*IST, to enable faster coding.

A general review of the interview transcripts confirmed that the survey instrument worked well. It was found that interviewees can be prompted to think of a critical incident, and the variety of incidents encountered was encouraging for the main phase aims of the Strand A. The critical success factors element was successful. This user-centred approach provided a neater and probably more valid approach to gaining a picture of the present and future pattern of EIS use than asking directly for awareness and frequency of use of particular services.

The interviews themselves were productive, but not time-consuming, generally taking around 20-30 minutes. Interviewees did not themselves note any problems. The approach produces the required degree of empathy between interviewer and interviewee, and the variety of topics discussed ensured that the interviewer did not suffer from survey fatigue.

Students were often slow to respond – generally slower than staff. To increase the response (speed and quantity) an incentive was offered for the main phase subjects. Names of those interviewed were entered in a prize draw for record token (£15) prizes, the aim being to allow at least one prize per site.

There were a number of interviewers involved, raising the possibility of interviewer inconsistency. Analysis of the transcripts suggested that this is not a problem; in terms of their depth, similar types of responses were obtained.

This section outlines the procedures followed (Section 5.3.4.1), summarises the problems encountered, and measures taken as a result (Section 5.3.4.2). Plans to revise the access procedures for the second cycle are discussed in Section 5.3.5, where the piloting for that was done as part of the Exception Plan.

Prospective departments were approached in October/early November 1999 by e-mail (Appendix 6), with a similar e-mail sent to librarians and/or heads of information services at that time advising them of the JISC survey (Appendix 7). Queries were resolved with responding departments, with further follow-up of non-responding departments at the end of November, and another follow-up in early December.

Plans to ‘over-sample’ in case of refusals, and withdrawals, proved justified. Three institutions and departments were unable to participate for the stated reasons:

• one refused to participate (no reason given) (1 department only selected)
• one was unable to participate as the College is being split among two other HEIs
• one refused to participate (various reasons, but none concerning QAA, or organisational restructuring) (1 department only selected).

A further follow-up of non-responding departments was initiated in early January 2000.

The final list of contributing institutions covers all categories in the sampling frame but participation problems continued through to the stage of face-to-face interviews with staff and students. Where interviews were obtained these were generally very productive, but there were problems in getting students to commit themselves to a time for interview, and even after making arrangements there were several ‘no-shows’.

The decision was taken to use an alternative method for data collection and a blanket e-mail questionnaire (Appendix 11) which was a slightly shorter version of the survey instrument was distributed. This was successful in some institutions, but not enough to be confident about using this as a back-up approach if student participation at a site looked problematic. For some institutions it was easier to treat this as a postal questionnaire, and postal surveys were conducted at institutions where the e-mail coverage was not complete for all students, or where there were part-time, or full-time students who rarely used the university e-mail system. Variations were made to the survey instrument, depending on the institutional requirements, target group (staff or students), and experience gained from the initial round of replies where it was apparent that some respondents had misunderstood the instructions.

It was obvious at the sites visited was that some liaison or ‘promotion’ by staff (particularly academic staff) seemed to help in alerting students to the survey work. This does raise the problem of a biased sample, but in the event it seems far better to have a large response and allow for the bias rather than a pitifully poor response from a purely randomly selected sample (where that was possible).

Following discussions with the JISC Scientific Adviser, the JUSTEIS team proposed an approach for the next cycle which would make more use of academic liaison as a means of contacting students and other academic staff within selected departments, following problems noted at the Advisory Board meeting (Appendix 15), and discussed at the regular team meetings as noted in Appendix 16.

These problems have not been widely reported in the literature although it is noticeable that there are few studies of student users across many HE institutions, and of those, one UK project (TAPIN) indicated that students are hard to recruit (Reid and Foster, 2000, p128). Action taken for the main phase had included the monetary incentive of inclusion in a prize draw but this had not noticeably increased response.

In addition, the number of departments who had withdrawn or were otherwise problematic in their participation in the survey tended to be concentrated in one or two discipline clusters. A meeting with the JISC Scientific Adviser on 15 March 2000, informed by discussions with the JUBILEE team on their experience, considered the measures already taken to increase response (e.g. e-mail and postal questionnaires, incentives). Following this meeting the JUSTEIS team was invited to present a plan to pilot a proposed method of contacting students through liaison with academic staff. Both JUBILEE and JUSTEIS had found, independently, that the various means of staff promotion of the survey work did appear to increase student response.

The JUSTEIS Exception Plan was instigated in March 2000.

A proposal requesting a two-month extension was formulated and submitted to JISC, and subsequently permission to undertake the extension was granted (Appendix 3).

The strategy was to identify academic staff and conduct a preliminary briefing within each institution. This approach was adopted in favour of an earlier suggestion that staff should be briefed at a regional meeting, as there were significant administrative difficulties in bringing together academic staff.

Three HEI sites were selected as a cross-section of types of institutions in the main survey, and as their proximity to Aberystwyth facilitated the easy access required for the short time frame of the extension. Within each, two academic departments were chosen for interviews and questionnaires; a third department was chosen for the questionnaire survey only. Departments were chosen to reflect the main survey sample and to try to compensate for deficiencies in some discipline clusters (Appendix 17). No student sampling was imposed by the research team; the choice of students being left to the local academic staff. Inevitably, this is not necessarily random sampling.

A member of the library or administrative staff was selected within each institution, to make initial contact with an appropriate member of the academic staff in each of the selected departments. Meetings were arranged between the selected academic staff and members of the research team, in order to explain more fully the objectives of the project, and the implications for the staff themselves. All staff were enthusiastic, perceived the relevance of the research and were willing to help. This contrasted strongly with some attitudes encountered on the main survey. It is clear that the face-to-face meeting between the academics and the researcher were essential to enlisting support and cooperation.

Each of the academic staff was asked to nominate five students for personal interview by one of the researchers, and also to arrange the completion of either hardcopy or e-mail questionnaires by at least 25 other students in their department. The staff member of the third department distributed 30 questionnaires. In fact, to facilitate this number of returns and based on our experience in the main survey, in the region of 500 questionnaires were distributed.

Student interviews were arranged at times convenient to all over the following weeks. Academic staff selected five students from their department, and arranged an interview room, and a timetable for interviews. In two of the six departments, one or two students failed to attend the interview, but substitutes were found either by the member of staff, or the researcher; and in another department more than five students reported for interview. The final number of students interviewed was 32 (Appendix 17).

Despite several attempts to contact a third department in one institution, a response was not forthcoming. A list of departments who provided questionnaires is delineated in Appendix 17.

Some of the academic staff interviewed by the Research Team believed that, in order to get the best response from students, the preferred time to distribute questionnaires would be during a lecture, to be completed and returned at the end of the lecture. Four members of staff used this method, three distributed questionnaires via e-mail and one handed them out during a lecture and asked students to return them to central collection point in the department.

The practice used in the main survey of awarding a prize to participating students was followed, although this offer was limited to those interviewed.

The extension methodology resulted in a high response rate for both interviews (32), and questionnaires. A total of 255 questionnaire returns were received from the three institutions surveyed.

The high response rates would suggest that the use of academic staff as the primary access route has considerable advantages over the method employed in the main survey and should be the preferred approach for organising interviews and distributing questionnaires for the second cycle. This finding reflects that of the JUBILEE project (JUBILEE Annual Report (draft), section 6 page 8). As the academic staff are more highly motivated and committed to the project as a result of the initial meeting, they are, in turn, much more successful in engaging with the students and facilitating their cooperation. There is the problem, naturally, of biasing the sample and that is apparent in the analysis of the data, where a large response from one particular department at one time of the year (e.g. students doing projects and looking for jobs) produced some oddities in the analysis. Assuming that the responses from the institutions are all comparably high this is less of a problem.

In addition to the central raison d’être of the initial briefing meeting with academic staff, the session proved valuable as a mode of dissemination about the research project, creating an awareness about the activities of JISC and of the issues underpinning the use of EIS in HEIs. It should be borne in mind that the names suggested to the team by the library were, in some cases, IT or information enthusiasts, and that it will be necessary to try to engage other members of staff, not just the initial enthusiasts, or tame library users, in the second cycle. It does not relieve the research team entirely from the problems of HEI compliance with the Data Protection Act, but if the academic staff can ask students to volunteer, and ensure that the students are fully aware of their rights, then this could cut down the time required for such administrative procedures.

The academic staff of two of the HEIs were of the firm belief that it was more advantageous to hold the initial briefing meeting within the institution rather than regionally.

With respect to the timing of the interviews, the preferred period was November to early December, however, several staff noted that timing is likely to vary from department to department and from institution to institution.

There was ambivalence regarding the use of e-mail questionnaires. Two HEIs commented on the dearth of e-mail communications within their institutions, which reflects the views of other universities in the main survey. It seems likely that both hardcopy and electronic questionnaires should be employed in the second cycle. There was a consensus among the academic staff of the three institutions that hardcopy questionnaires are best distributed during lectures.

Analysis was governed by the type of data (interviews or questionnaires). Procedures are outlined in Section 5.4.1, where a breakdown of the site responses is given. Section 5.4.2 describes the approach taken for the qualitative analysis using the NUD*IST software package. Section 5.4.3 describes how the interview transcript extracts have been displayed in the report.

The thematic coding (termed nodes in the NUD*IST software used) was continuously developed. Two project team members were responsible for the coding, one doing the initial coding and the other verifying, checking and amending the coding to ensure that the provisional hierarchy represented the themes to be extracted (Appendix 14).

In view of the quantity of qualitative data (c.150 interviews for Strand A alone, with another 10 interviews of librarians’ purchasing intentions for EIS), tables and profiles were developed to enable data reduction and data display.

The quantitative analysis allowed a good indication of the scale of a particular trend or feature. Qualitative analysis explored some of the contributory and explanatory factors.

xyznn where xy = site code number

z = 1, 2, 3, 4 (1 = student, 2 = library staff (Strand A interviews), 3 = academic staff and 4 = LIS managers (purchasing interviews, Strand C)

nn = sequential number given to interviewee in the site category

The data analysis software has two main options for categorising input documents – by line or by paragraph and both were used. The position of each extract in the ‘document’ (interview transcript) is noted. In some cases the extracts have been condensed, and this should be apparent from the number of text units used for the extract, where the line system has been used. The data is anonymised but it is easy at a glance to see the range of institutions from which the extracts are drawn and whether the data comes from a student, or member of staff (academic or LIS).

---

This page last updated 25 August 2000 10:03:19
Research Pages maintained by Chris Armstrong