paper by H Layton

home > shllc2001 > papers > paper by H Layton

Rivers of knowledge

9th Specials, Health and Law Libraries Conference

Electronic resources: matching client attitudes and usage data to management decisions

H J Layton
Defence Science and Technology Organisation, Adelaide, South Australia

Introduction

The aim of the paper is to examine the process of making management decisions about electronic resources in a research library environment. Using the DSTO Research Library as a case study the methodology and outcomes of three client surveys about use of, and attitudes to, electronic resources are presented. The proposition that the key determinants of management decisions are client feedback and analysis of usage data is explored and confirmed. The result is a continuous loop that leads to informed management decisions against an ever-changing environment.

The findings and analysis of the client surveys may be helpful to other libraries as they make decisions about a model of service for electronic resources. An outline of usage data provided by vendors is given, as well as a description of the DSTO experience and a rudimentary review of the experiences of others in a similar environment.

Brief background of DSTO Research Library

The Defence Science and Technology Organisation (DSTO) consists of 2200 staff located at seven research sites across Australia, plus a smaller presence at a range of Defence establishments. The large majority of the Library's client base comprises highly specialised tertiary educated professionals. Of the 1900 staff in the research divisions, thirty-seven per cent have been awarded a doctorate (some 700 staff) and a further forty-three per cent are professional officers with at least a bachelors degree. The remaining twenty per cent are technical or information technology officers who frequently have qualifications ranging from certificate to degree level. In South Australia the Library has recently assumed the additional responsibility of service to 4000 Defence personnel.

The main subject areas covered are the sciences, engineering and information technology. Other subject areas include human factors, nutrition and biological warfare.

DSTO Research Library operates under a corporate structure with five staffed libraries supported by substantial hard copy collections and librarians at two other 'remote' sites which do not have substantial hard copy collections. The emphasis at the remote sites is on access to information. Hence the DSTO Research Library is experimenting with two different models of service and like many other libraries we are trying to implement the right balance between hard and soft copy. The Library employs thirty-seven staff in approximately thirty three full-time equivalent positions.

The environment

The environment in which research libraries make decisions about the purchase of electronic resources includes the following components:

strategic planning. Decisions must be linked to the organisation's future directions and business plans;
senior management. The demands and expectations of senior managers may be a driver for changing models of service delivery;
budget. Many libraries are working in an environment with static, or reducing, budgets. However some libraries have been successful in obtaining special grants in order to provide electronic resources, by arguing that the investment will yield dividends with an increase in use. Demonstrating a link to increased productivity is a key factor;
cost benefit analysis of print versus electronic copy. There are a range of pricing models for print and electronic products. Links to usage and productivity are essential;
license agreements. Libraries need to clearly define in any contracts variables such as definition of the user population, sites, IP address and interlibrary loan provision;
consortia or go it alone? Where possible, libraries should initiate consortia arrangements to share costs and increase purchasing power. Membership also gives member institutions stronger bargaining power over the terms of a license agreement; and last but by no means least,
client attitudes and needs.

Client feedback

How do you find out what the client wants? At DSTO we conducted a range of client surveys which enabled us to build up a picture.

In June-July 1999 a series of exit surveys were conducted. We surveyed 915 clients in four libraries. We asked six quick questions of clients leaving the library - three questions relating to the visit and three general questions about library use. The questions could be answered in two minutes.

This was a comparatively low cost exercise. The material obtained was very useful for 'managing upwards' for example, in answering the perennial question 'with the internet, do people still visit the library?' The answer was a clear 'yes' - there were 1078 visits to the four major libraries in a week. We obtained information on the number of repeat visits, the amount and level of client training required, the level of satisfaction with service, why people visited library and the quantity and level of assistance sought.

Obviously this type of survey captures staff in the organisation who use and visit the library - it does not capture the non-library users.

All DSTO staff have desktop access to networked library resources. At this stage we were offering desktop access to the library catalogue, a number of major databases (some in full-text, some in abstract form) and about 120 journals in full text. The journals comprised eight per cent of all serial titles. The final question in the survey was 'What services do you use from your desktop?'. We found that seventy-seven per cent of clients who were surveyed accessed the library and searching service tools from the desktop. This told us that clients were both visiting library and accessing its resources from the desktop.

We found that many clients were using a range of our resources from their desktop. Clients volunteered a range of informal comments which led our surveyor to conclude 'desktop use seems to be increasing' and 'many clients seem to feel pressured into using desktop services'. [1] This survey was a good starting point.

In November-December 1999 we conducted a review of the library WebPage. Using an electronic form the aim was to gather client and library staff feedback on the Library WebPage and to make recommendations for a redesigned page. Since the page had been developed, the library had increased its commitment to electronic resources.

The survey comprised seven questions. Eighty-five respondents told us they used the WebPage to access the catalogue, databases and electronic journals in that order of priority. Our findings were generally consistent with other studies. We found as did Rogers, that levels of use for databases were higher than for electronic journals, because databases had been available for a longer period of time. [2]

The results told us that the catalogue, databases and electronic journals were appropriate targets for continued expansion. Clients also told us they wanted easier access to electronic journals with an improved interface. We were starting to build up a picture.

As with the previous survey, some of the most valuable data was the informal comment - clients revealed frustrations with passwords and with the limited electronic journal coverage we were offering. There were also difficulties in finding the required information on the WebPage.

In May-June 2000 DSTO Research Library conducted a Client survey on electronic journals. Issues associated with electronic journals had been presented to senior managers and senior clients for consideration. Library managers now needed to take a strategic approach to the expansion of the electronic journal service. To date, emphasis had been on the provision of hard copy journals with some supporting electronic versions where these were available at no additional cost.

The aims were:

To identify usage patterns, user preferences and barriers to use
To provide specific information about titles and subject interests
To collection information from current users of electronic journals as well as potential users
To determine if some subjects were particularly suited to electronic delivery and if some were suited to the print version.

The key question to be answered was: 'Is there a link between productivity and electronic journal use?'

We conducted an extensive telephone survey with 210 interviewees. There were forteen questions. Some were closed; some were multiple choice, others were open-ended or required answers ranked by priorities. Development of the questionnaire and the survey sample was a complex process including a trial survey which enabled us to make adjustments. The survey sample was considered representative and data was considered statistically viable.

The main findings are as follows: [3]

Forty-seven per cent of respondents currently used electronic journals. Library staff felt that this figure may be inflated due to confusion between the terms 'database' and 'electronic journal'.
The most valued aspects of electronic journal usage are reliable access and comprehensiveness, followed by the ability to download and concurrent searching across a range of years[4].
Respondents prefer to use hard copy for browsing and reading, and electronic journals to search for information.
Respondents expressed a high level of concern about technical problems - connections dropping out, slow downloading etc. Users wanted assurance about connectivity.
Fifty-eight per cent said they were willing to forgo hard copies but of these only fifteen per cent gave unqualified support. There were many issues to be resolved first. Common concerns were the accessibility of electronic journals and the need to download and print. The research gave no clear endorsement for the discontinuation of hard copies.
Sixty-two per cent were concerned about possible lack of archival access when converting to electronic journals
Clients identified on-line tutorials as the most useful type of training in electronic products
Client confusion about table of contents services abounded; they were also unclear about the distinction between databases and electronic journals.

As an aside, there were, and continue to be, issues for us relating to how we organise and present our electronic resources.

Additional comments provided by respondents were valuable albeit wide-ranging. They varied from 'I am strongly against hard copy being cancelled for e-journals' to 'I would like the library to obtain CD-ROMs as backup' to 'I wish them good luck'.

We needed more than 'good luck' to make informed management decisions!

Analysis

The results showed that our clientele lacked an understanding about electronic journals. The format was not as well developed as clients seemed to think. Adams and Bonk in their survey of university facilities found that the most common obstacle to the use of electronic information was lack of knowledge about resources,[5] followed by lack of training.

Rather than take a generic approach, we asked staff with access to hard copy collections which hard copy journals they regularly use that they would prefer in electronic format. We were trying to determine which electronic journals would adapt well to the electronic environment. There was no clear outcome. Highlighted was some consumer resistance to the electronic version and a need to track usage.

In her study, Rogers found that high numbers of graduate students (thirty-three per cent) and faculty staff (twenty-four per cent) in a university environment were unable to articulate any disadvantages in favour of using electronic journals. [6] Our researchers on the other hand certainly had no trouble in articulating the disadvantages! Roger's respondents also reported concerns with technical issues such as connections.

DSTO's study was conducted too early to make a valid assessment of users' experience. It was instead a survey of client expectations based on rather limited use and an assessment of their readiness to further utilise electronic resources. In summary, the issue of electronic delivery is framed by the capacity to measure what clients use, how often they use it and the way it is used. Only by answering these variables can we look for a link between productivity and electronic resource usage.

DSTO response to client surveys

Providing both print and electronic journals is frequently more expensive than one format only. We conducted an analysis of a sample of our print journals and found we would need an additional fifteen per cent expenditure to purchase both. This figure is in line with the 'most common' percentage surcharge for electronic journals of fifteen to twenty per cent provided by Swets Blackwell in their 1999 survey of publishers.[7]

DSTO's surveys showed that clients were receptive to electronic journals but harboured concerns. We needed to offer a critical mass of electronic journals to entice our clients to further change their methods of accessing journals. The drivers were the needs of remote site users and the desire of senior managers to be matching developments in other organisations.

We were grappling with demands from users at remote sites for the electronic version of a particular database which included electronic journals, so in our Client Survey on Electronic Journals we asked a series of questions specific to that product.

Sixty-five per cent of those who used the product in hard copy format indicated they were willing to forgo the hard copy if an electronic copy was available.

This was considered sufficient support to implement a change and the sample size large enough to provide a sound basis for planning. The answers and a two week trial enabled us to make a management decision on that product. We cancelled the hard copy and purchased the electronic.

The conditions of license agreements were important to DSTO Research Library, as we needed to offer products to remote site users. In all the library purchased subscriptions to several electronic products, offering a far wider range of full-text resources than existed for our hard copy subscriptions. In 1999 DSTO Research Library offered 120 electronic journals in full text; two years later we offer 723 journals in full text. Numbers of hard copy journals are reducing gradually and the trend will continue.

Usage data

Usage data enables a library to assess the effectiveness of the use of a resource and to enable librarians to identify information needs and over time to analyse trends. Mercer makes the case that librarians must be able to measure digital use of electronic journals and books in order to make the best purchasing decision for their institutions.[8] Librarians also use usage statistics to enable us to identify a need for training and marketing and to plan future directions.

Ideally the library manager wants to find out the following:

the number of titles accessed;
the frequency of access;
any problems with log-ins (to assess whether you have purchased enough access) or connectivity;
does usage show demand for 24-hour, seven days per week access?
the number of downloads;
the frequency and effectiveness of search capacity;
the use of Help services;
in a multiple site organisation, identify site use (do those with access to a substantial hard copy collection have different patterns of use than those with minimal hard copy access?);
if archival access is needed and if so, how far back; and
the impact of Table of Contents services on hard copy and electronic journal access.

Of course usage statistics for hard copy journals also apply to the subscription review process. Librarians must meaningfully measure statistics for both print and electronic resources, and in a way that ensures valid comparison. It is beyond the scope of this paper to provide solutions, but there are a number of proposals in the literature. [9] Early experience suggests that user statistics for electronic access are more accurate than for the print format.

What do vendors supply?

There is a great range in the type and quality of usage statistics provided by vendors and of course they vary according to the product. The quantum ranges from none at all to comprehensive analysis received by personal e-mail each month. Some suppliers will e-mail usage data to the library at intervals (often only when prompted); sometimes library staff can access data from publishers' web servers. Statistics may be reported monthly, quarterly, every six months or simply irregularly. Sometimes it is necessary to obtain a definition of a statistic - what does it mean? Statistics may include number of searches, length of search time, downloads, log-in fails, journal title hits, page hits, most popular titles per month etc.

Inconsistent standards of reporting amongst vendors make it difficult for librarians to report usage to senior managers, clients and library staff in a uniform way. Librarians frequently re-package the information in order to present it in an attractive and relevant manner.

Both libraries and vendors are at an early stage in the provision and analysis of usage statistics. The Swets BlackwelI 2000 e-journal survey of thirty-three STM publishers showed that only forty per cent of respondents provide usage statistics.[10] Vendors and librarians have yet to get the balance right - we need relevant information but not too much of it. It is up to librarians as clients to inform vendors of our requirements and to work with them.

The Association of Research Libraries (USA) has completed Phase 1 of a study to determine usage measures for electronic information resources. Findings are that a number of research libraries have developed useful strategies for developing such measures but that it is too early to offer 'best practices'. The study shows that 'libraries are experiencing serious information management problems due to inconsistent and non-comparable statistics supplied by database vendors'. [11]

Usage patterns - the experience of other libraries

Not surprisingly, an examination of the literature reveals an increased use of electronic journals and decreased use of printed journals over the past few years.[12] It must be noted that the samples are usually undergraduate students, professional students or faculty staff at university libraries in the USA. Tenopir and King have made a detailed study of the use of electronic journals by scientists in the USA and confirm the general trend.[13]

A number of writers report that electronic journal use is different from that of print journals and that we are facing a general change in the scientific communication process. [14] In his case study of the transition from print to electronic journals McLean[15] reported the following usage patterns:

A significant number titles are of little or no use.
Based on a transactional cost of US$15, only forty-four per cent paid for their subscription value, that is. librarians cannot be locked-in to historical views of relevance in terms of journal title selection.
31.5 per cent of usage was from unsubscribed titles.
Unrestricted access encourages a much wider use of titles and is likely to generate a much higher number of annual transactions.

McLean's findings support a rethinking of the assumption that the subscribed list of titles be used as a basis for a pricing model.

Some librarians warn that through journal aggregations libraries are purchasing electronic journals they neither need nor want.[16] Hence usage statistics may be particularly relevant when assessing an online aggregation of journals to ensure that we are buying access to the journals our clients need.

The DSTO experience

At DSTO we are using a number of different models to offer electronic resources. Without exception we are observing heavy take-up of electronic resources available at the desktop with Library Managers reporting usage to the organisation at every opportunity. It will be time soon to repeat our earlier surveys in order to formally assess user satisfaction and to look for trends and links to increased productivity.

Where we are duplicating journals in hard and electronic copy as part of a subscription package we are monitoring activity statistics for both online and print journals and using them as one of the tools of the subscription review process. Client feedback is also an integral part of the review process.
One package of products includes full text access to 317 electronic journals, of which 26 titles are duplicated in hard copy form. A detailed analysis of the database shows that our users are accessing a far wider range of titles than we subscribe to in print format:
- four of the Top 10 journals were titles to which DSTO did not subscribe in print form.
- six of the Top 10 full-text articles were titles to which DSTO did not subscribe in print form.
- each user session averaged five searches.
- two to three full-text articles were downloaded on average each session.
- of the 317 titles available, users accessed 158 in their searches. Only twenty of these were titles to which DSTO subscribed in hard copy form.
In databases where both full text and abstracts are available, there is a strong trend for users to go direct to full text searches for viewing and downloading. This indicates satisfaction with downloading times.
Usage statistics have enabled us to monitor concurrent usage for any bottlenecks. In one instance there were an unacceptable number of 'failed log-ins' and we increased the number of user licenses.
When checking and (negotiating) license agreements for electronic information it is vital to look for a clause stipulating that the license provides for usage data. We found that usage data was not available for one of our major products and changed suppliers.
DSTO Research Library has experienced some negative feedback from researchers about replacing the hard copy with electronic format. Comments are consistent with those recorded in the Client Survey on Electronic Journals. On the other hand the majority of clients have given positive comment and remote site users are elated about the new electronic products. Not surprisingly, remote sites show heavy pro rata use of electronic products.
We have experienced the operational advantages of providing journals in electronic form - reduced staff processing time and savings in storage space. However maintenance of electronic resources, journals in particular, is time-consuming and requires a different set of skills.

Conclusion

Library management decisions about the take-up of electronic resources must be based on client feedback, with subsequent monitoring dependent on the provision of relevant usage data. Usage statistics for electronic journals in particular are a vital performance measure and show different patterns of use from that of print journals. Librarians are urged to clearly articulate to vendors the performance measures they require and to work with vendors until they are provided. Finally client usage should be reported to senior management, clients and library staff.

Bibliography

Adams, JA and Bonk, SC (1995) 'Electronic information technologies and resources: use by University Faculty and Faculty Preferences for Related Library Services'. College and research libraries, 56, March 1995, 119-131.

Association of Research Libraries, USA. (2000) 'ARL E-metrics project: developing statistics and performance measures to describe electronic information services and resources for ARL libraries. Phase One report.' http://www.arl.org/stats/newmeas/emetrics/

Bauer, K. (2001) 'Indexes as tools for measuring usage of print and electronic resources'. College and research libraries, 40, January 2001, 36-42.

DSTO Research Library (2000). 'Electronic journals: Issues Paper'. [First draft prepared by Donald Keast; final paper presented by Heather Layton].

Frazier, K. (2001) 'The Librarians' dilemma: contemplating the costs of the 'Big Deal'. D-Lib Magazine, 7(3), 1-9.

Kristine Peters Project Management (1999). 'Defence Science and Technology Organisation. Results of Exit Survey - client use of the Library.'

Kristine Peters Project Management (2000). 'DSTO Research Library: Client survey on electronic journals'.

Liew, CL, Foo, S and Chennupati, KR (2000) 'A study of graduate student end-users use and perception of electronic journals'. Online information review 24 (4), 2000, 302-315.

McLean, N. (2000) 'The transition from print to electronic journals: a case study'. Paper presented at: Sustainability in the Scholarly Information Market Place: the Search for Digital Library Business Models. 2 May 2000. http://www.lib.mq.edu.au/conference/sustainability/transition.html

Mercer, LS (2000) 'Measuring the use and value of electronic journals and books.' Issues in science and technology librarianship, Winter 2000. http://www.library.ucsb.edu/istl/00-winter/article1.html

Montgomery, CH and Sparks, J (2000). 'Framework for assessing the impact of an electronic journal collection on library costs and staffing pattern'. http://www.library.drexel.edu/facts/staff/dean.html

Rogers, SA (2001) 'Electronic journals usage at Ohio State University'. College and research libraries. 40, January 2001, 25-34.

Swets Blackwell (2000) 'Swets Blackwell's 2000 e-journal survey'. Homepages, Spring/Summer 2000, 10-11.

Tenopir, C and King, DW (2000) 'Towards electronic journals: realities for scientists, librarians and publishers.' Washington, D.C.: SLA Publishing, 2000.