EiS Surveys JUSTEIS: JISC Usage Surveys: Trend in Electronic Information Services


The JUSTEIS Project Methodology Overview

Introduction
The Joint Working Group on Use and User Behaviour convened following the report by Day, East and Stone (1997) highlighted a number of key considerations in
its report which it was hoped would lead to work at the integration rather than innovation end of the spectrum. The considerations listed were:

  1. Funding and Budgeting;
  2. Regulatory and Technical Restraints;
  3. Quality of Content and Service Quality;
  4. Skills, Awareness and Training; and
  5. Broader Issues of Institutional Mission and Strategy.
The feeling expressed was that the uptake of EIS within HEIs was influenced by both internal (history and mission) and external (cost and availability) factors and that it would be appropriate for JISC to gain a knowledge of the awareness, use and availability of EIS such that JISC could become a "learning system" for HEIs. Accordingly, the JISC Circular 1/99 is framed within CALT/CEI and Tavistock recommendations and the proposal should also take them into consideration.

In order to understand and address the needs of scholars, teachers and learners in their use of EIS, JISC wishes to monitor and evaluate the behaviour of users in information seeking and use of information technology and information services. A range of quantitative and qualitative survey instruments has been proposed to determine actual use and declared user needs.

  • Work area A involves a single (but differentiated) broad-based sample survey of patterns of use and non-use of all EIS by librarians, academics and students. This is to be administered through face-to-face or telephone interviews so as to ensure accurate sampling of all relevant sub-populations (roles and disciplines).
  • Work area C will determine resource access and provision by UK HE; the methodology suggested by JISC is a Web survey backed up by face-to-face interviews.

The Steering Group for this proposal saw a logic in linking work tasks A and C. Specifically, it was felt that this had certain advantages:

  • The use of consistent definitions and a unified taxonomy of EIS for both surveys;
  • Methodological advantage from the construction of common survey instruments;
  • Resource savings made through optimal use of manpower during data collection; and
  • The ability to cross-tabulate data between tasks.

Stages
The creation of a clear definition of what constitutes an EIS and the development of an EIS taxonomy is an important step before the surveys can begin. In order to form a thorough picture of EIS provision and use, the team believes that the following resource types/services should be considered, together with such others as may emerge during discussions:

  • Subject gateways (EEVL, OMNI, SOSIG, BUBL, NISS, etc);
  • JISC/CHEST negotiated services (BIDS, OCLC, EDINA);
  • Other single host/multiple user services (traditional online vendors and others such as free MEDLINE services);
  • OPACs and clumps (COPAC);
  • University Web sites/Campus-Wide Information Services;
  • Current Awareness Services/Individual Article Supply and other electronic document delivery services;
  • Messaging Services (E-mail discussion lists; Mailbase archives; USENET news);
  • Data sets (University of Essex Data Archive, MIDAS);
  • Text archives (Oxford Text Archive, Project Gutenberg, University of Virginia Electronic Text Centre);
  • Electronic journal collections (JSTOR, Ingenta, NESLI);
  • Individual publisher sites (e-journal and e-monograph);
  • Other Web electronic information resources (dictionaries, term banks, encyclopaedia, thesauri);
  • Locally mounted electronic resources such as CD-ROMs;
  • Electronic Collection Management Services.

Working against the prepared taxonomy, work in area A will need to:

  1. Determine the scope of possible EIS available to HE institutions.
  2. Determine perceived user needs for each population for a range of academic disciplines.
  3. Determine user awareness of EIS availability.
  4. Sample actual use by each population for the same range of disciplines.
  5. Determine the degree of satisfaction with available EIS.
  6. Determine reasons for non-use.
  7. Determine reasons for non-provision/subscription.
In work area C preliminary investigations suggest that the online review of EIS availability as documented on HEI Web sites will show a wide variation in levels of detail so that limiting the survey to this approach would be inadequate. HE Web sites may not include, for example, CD-ROMs issued to users from a readers' enquiry desk. It seems to the steering group that the methodology would need to be supplemented by an additional survey.

Populations
The project would use cluster sampling and secondary sampling with probability proportional to population size for student, academic and LIS staff within the cluster. The clusters would be based on the RAE Units of Assessment for 2001. Sampling would involve randomly selecting 30 departments (covering single site 'campus' universities, institutions with separate autonomous information services and those with franchised courses, distance learning provision, remote campuses, and both traditional and new universities); taking a random sample of, say, six departments in each of five discipline clusters (social and health sciences; humanities; engineering, art and design; physical, biological and clinical sciences; and business studies); and selecting 100 academics (covering all professional grades: professor, reader, senior lecturer, lecturer and academic-related staff) and sufficient students from each group to ensure that there are approximately equal numbers of first, second and third year undergraduates (both full- and part-time) as well as a smaller numbers of taught postgraduates and research students (probably using probability proportional to the size of the total population in the UK). Approximately 250 LIS senior/middle management staff would be randomly selected across all chosen HEIs. This would give a potential sample of 5 subject discipline clusters drawn from 30 departments; selecting 100 academics and 150 students from each discipline cluster would, with 250 LIS staff, give a target of 1,500 individuals.

Methodology
Four principal methodologies would be used to gather data for both work areas. It has been found that the use of casual student labour to undertake telephone surveys is not satisfactory and therefore this proposal is based on the use of experienced staff and one full-time research assistant.

  • Critical success factors (CSF) (Work area A.)
    The approach would involve asking the users what information was important to their work, the factors critical to the success of work objectives and the information required to support those critical success factors. This is an accepted methodology for this type of work as seen, for example, in the investigation of the management information needs of heads of academic departments in universities in the United Kingdon (Loughridge, 1997). This survey will form a useful baseline for the other methodologies and serve as a cross-reference for their findings. There will be preliminary pilot study to test out the survey instruments.
    The CSF survey would examine the following populations: students (taught); research students; academic and academic-related staff; and the LIS-as-user.
  • Critical incident technique (CIT) (Work area A.)
    Critical incident technique would involve an interview asking users about a recent information-seeking incident that they can recall easily. This can be used to explore the reasons for seeking information, the sources used (and reasons for preferring particular sources), time spent, effectiveness of search results, and what will be done with the information obtained. The interviewer would also widen the scope by requesting a list of sources used in the last three months. Again, there will be preliminary pilot study to test out the survey instruments.
    The CIT survey would also examine the students (taught); research students; academic and academic-related staff; and the LIS-as-user populations.
    In this context, the technique might reveal awareness (or lack of it) of certain sources of information, the pattern of use of sources, and apparent reasons for success or not. It would begin to explore the non-use but the interview would need to emphasise that incidents that were not followed up, or 'failures' were also of interest.
    The steering group is aware of a useful taxonomy of value proposed by Saracevic and Kantor (1997) which, it is felt, could be used to develop a framework of questions concerning reasons for using an information service (related to task, personal reasons, acquisition of an 'object' (e.g. document) or performing an activity (browsing); and aspects of interaction and results of service use (value estimates).
    Critical incident technique is an appropriate means of exploring the patterns of use - matching types of sources used to purposes, and looking at user group differences in patterns of need and use.
  • Web survey (Work area C.)
    In order to establish availability of EIS within HEIs, an online review of resources documented on HEI Web sites will enable a wide sampling. The approach would involve examining the information services/library areas of the Web site and documenting the listed EIS. The listings would be delineated according to the established taxonomy.
    As this survey involves desktop research and does not involve face-to-face or telephone interviews, it is possible to encompass a larger sample. The Web survey will examine the LIS-as-provider population in all HEIs.
  • Supplementary structured interviews (SSI) (Work area C.)
    Structured interviews would be used to supplement the Web survey and would examine the same user population. The survey instruments will be piloted before use. The sample of 20 to 25 HEIs would be identical to that used in Work area A and would be tested to determine the completeness, accuracy and currency of their listing on their Web pages. The data would also cross-tabulate with the user perceptions of sources used in the last three months from work area A.

Back to Top of Page

© JUSTEIS Project Board, 1999
This page last updated 15 July 1999 14:49:08
Site maintained by IAL
 

Top
Work Tasks
Populations
Methodology

Back to Home Page