paper by Grimes, Ruthven and Ritchie

home > shllc2001 > papers > paper by Grimes, Ruthven and Ritchie

Rivers of knowledge

9th Specials, Health and Law Libraries Conference

What makes a top-performing interlending operation: the ILL/DD Benchmarking Study and Special Libraries

Sue Grimes
librarian, Nepean Hospital Library
Tom Ruthven
interlending services director, National Library of Australia
Kristen Ritchie
project officer, Benchmarking Study National Resource Sharing Working Group

Introduction

In recent years there has been increased focus on the performance of libraries in delivering high quality interlibrary loan (ILL) and document delivery (DD) services to their patrons. The internet and electronic journals have increased patron expectations about delivery times and copy quality.

The National Resource Sharing Working Group was established in 1999 to look at ways of improving the efficiency, reliability and client satisfaction of ILL/DD operations in Australian libraries. The National Library set up the Working Group with support from the Australian Library Information Association (ALIA), the Council of Australian State Libraries (CASL) and the Council of Australian University Librarians (CAUL). Its members are ILL practitioners from a range of library sectors.

Over the last twelve months, the working group has taken a two-pronged approach to improving ILL/DD services in Australian libraries. It developed performance standards in the form of a new Australian Interlibrary Resource Sharing (ILRS) Code that was released in February this year. It has also undertaken a major study of ILL/DD practice and trends in Australia to identify best practice characteristics for benchmarking and training purposes.

This paper describes the methodology used in the Benchmarking Study, lists the draft results, and discusses the implications of the results.

The three key aims of the study are to:

identify the characteristics of high performing ILL/DD operations,
be an instrument for raising awareness and changing practices, and
assist any library to benchmark their operations against a standard set of data.

Methodology

The Study is the largest and most comprehensive study of ILL/DD operations ever undertaken of Australian libraries. The survey instrument was based on the Association of Research Libraries (ARL) study, Measuring the Performance of Interlibrary Loan Operations, undertaken in the United States with North American research libraries. The ARL Study was chosen as it was a successful study that had a detailed and proven methodology. Mary Jackson, who undertook the ARL study, is acting as a consultant to provide advice on the Australian Study.

The survey collected information in four areas:

general characteristics of the ILL/DD unit (a 17 page questionnaire),
direct cost for a full financial year, including staff; network and communication; delivery, photocopying and scanning; office supplies; equipment; software and maintenance; and fees and charges paid and payments received,
transactions and fill rate for a full financial year;
turnaround time for items requested by users in October and November last year, and
user satisfaction with the service/items supplied for these requests.

Libraries from all sectors participated. A request was made to University and National/State libraries for all organisations to participate. A call for volunteers from the special, public and TAFE sectors was also made. In all ninety-seven libraries provided data. Ninety libraries had sufficient data to be included in the bulk of the analysis:

Universities: 32
State and National Libraries: all
Public libraries: 20
Special libraries: 30

The survey was run primarily in October/November last year. Each general characteristics questionnaire and cost sheet was checked to ensure the data supplied was accurate and comparable. The turnaround time and user satisfaction survey forms were checked as they arrived. Participants in the Study were contacted where clarification was needed.

The Australian Bureau of Statistics was contracted to analyse a significant portion of the data. The Working Group and National Library staff undertook further analysis to put the results into context and to compare the characteristics of the highest performing libraries.

The final report of the Study is planned for release in September 2001. Each participant in the Study will also receive an analysis of their own operations that will indicate their performance against each of the performance measures.

Draft results

Requesting operations

The key performance measures used to compare library performance for requesting items were:

turnaround time,
fill rate, and
unit cost

The draft results of the study show that high-performing requesting operations had the following characteristics:

Patrons request items electronically.
Minimisation of the time a request was handled by their staff (for example, request went from the patron to the ILL/DD unit to suppliers very quickly).
A union catalogue/s is used to find sources of supply.
Items are requested electronically from supplying libraries whenever possible.
More than eighty per cent of filled requests are filled by the 1st supplier.

In addition, the use of reciprocal agreements was also a common factor in high-performance. The features of a reciprocal agreement that are important for effectiveness and efficiency include:

there are no fees charged by/to libraries participating in the arrangement;
information about holdings is easily and electronically available;
there is a joint electronic system for requesting and tracking requests;
the largest percentage (or a majority) of requests are filled by libraries within the agreement;
transaction numbers within the arrangement are fairly high;
there is a defined supplier group for the majority of requests about whose collections ILL/DD staff in requesting libraries have a good knowledge; and
there is a strong sense of co-operation between the libraries in the agreement.

The following table provides a summary of the main performance measures for requesting.

Table	Summary of performance measures
REQ	Turnaround Time (days)				Fill rate (%)				Total unit cost ($)
	10th	Av	Med	90th	10th	Av	Med	90th	10th	Av	Med	90th
Overall	3.0	11.5	9.0	23.0	77.1	89.2	91.7	99.1	$11.72	$32.10	$26.17	$63.96
National/State		12.5	10.0			79.1	88.3			$79.99	$76.69
University		10.8	8.0			89.3	90.3			$35.21	$29.19
Special		8.7	7.0			95.8	96.9			$24.76	$18.51
Public		16.6	13.0			83.3	85.2			$18.98	$20.33

Average performance did vary by sector. There were also significant variations between the highest and lowest values overall.

Turnaround time was measured as the number of days elapsed to complete the request from the patron's point of view i.e. from the date on the patrons form to the date the patron was notified of the result of the request. All completed requests (that is, filled and unfilled requests but not those 'in process') were used to calculate average turnaround time for the survey period. The time period covers elapsed days not just working days.

Fill rate was calculated from the requesting transaction figures for filled and unfilled requests over a full year that were provided by libraries in the General Characteristics Questionnaire.

Unit cost was derived from the total cost identified in participating libraries cost worksheets, divided by their total filled transactions for the year for requesting.

Supplying operations

The main performance measures used to compare library performance for supplying items were:

fill rate, and
unit cost.

The draft results of the study show that high-performing supplying operations had the following characteristics:

a majority of requests received electronically (eg. KDD, ILANET, OILLS, e-mail, online form);
a large proportion of their collection reported on a union catalogue;
an automated ILL system for receiving and tracking requests; and
a high percentage of requests filled through reciprocal agreements.

The number of times library staff check the shelves for an item (1, 2 or 3 or more times) had no effect on either fill rate or total unit cost.

The following table provides a summary of the main performance measures for supplying.

SUP	Fill rate (%)				Total unit cost ($)
	10th	Av	Med	90th	10th	Av	Med	90th
Overall	66.4%	82.9%	85.1%	97.3%	$5.84	$17.03	$14.54	$30.17
National/State		87.8%	90.8%			$20.13	$17.15
Public		79.3%	85.1%			$16.21	$13.68
Special		90.8%	90.7%			$15.20	$13.21
University		76.6%	78.4%			$18.47	$15.96

Fill rate was calculated from the supplying transaction figures for filled and unfilled requests over a full year that were provided by libraries in the General Characteristics Questionnaire.

Unit cost was derived from the total cost identified in participating libraries cost worksheets, divided by their total filled transactions for the year for supplying.

General characteristics

The Study collected a large amount of information about current practices and organisation of ILL/DD operations in Australia. For example:

Centralisation	87% of participating libraries categorised themselves as centralised requesting operations and 84% as centralised for supplying.
Charges to patrons	Only 28% of participating libraries never charged their patrons for requesting, only 13% never charged other libraries for supplying.
Staffing	On average, 89% of staff were categorised as ILL/DD staff, with the other 11% being staff working primarily in other areas - the average percentages were almost identical for requesting and supplying.
Transactions	The total number of transactions undertaken by the participating libraries in the surveyed year was 440, 827 for requesting and 540, 157 for supplying.
Loans vs. copies	On average, 58% of requests made to other libraries were for loans. 54% of items supplied by participating libraries were loans.
Collection size	The total estimated collection size for the participating libraries was 28,887,648.
Ariel	40% of participating libraries use Ariel for some part of their service.
PCs per staff	On average the participating libraries have 1.71 PCs per full time equivalent (FTE) staff for requesting and 2.04 for supplying.
Reciprocal agreements	68% of participating libraries select supply sources in the first instance on the basis of reciprocal agreements.
Income	60% of ILL/DD units retained some or all of the income generated from supplying.
Commercial suppliers	8.4% of participating library requests overall were supplied by Commercial Document Delivery suppliers.. Universities supplied the largest percentage of requests to the participating libraries on average. Were universities also the largest suppliers so that much of the traffic measure was uni - uni simple because of their volume?

Libraries were also asked about major changes in their ILL/DD operations in the past three years. It is interesting that:

Review of operations	38% of participating libraries indicated they had reviewed, streamlined or restructured their ILL/DD unit in the past three years;
Automation	39% of participating libraries had automated all or part of their ILL/DD process in the past three years;
Upgrading equipment	36% had upgraded their computer and/or other equipment (such as Ariel) in the past three years; and
Commercial suppliers	39% indicated they had significantly increased their use of Commercial Document Delivery suppliers in the past three years.

Costs

Total staff unit cost was the largest proportion of total unit cost for most libraries: 61.2 per cent on average for requesting activities and 76.8 per cent for supplying. Fees were the second largest unit cost category on average for requesting at 24.3 per cent, and delivery unit cost was generally the second largest category for supplying activities at an average of 11.3 per cent.

Other cost categories - such as equipment and network cost - have a marginal effect on total unit cost. This means that spending money to improve systems and processes would improve services without the need to increase staffing or increase other costs significantly. It should also be noted that staff and fee costs cannot be considered in isolation. Reducing staff costs and fees without making improvements to systems and processes runs a high risk of diminishing overall performance and patron satisfaction.

The data also showed that the unit cost decreases the more requests you process when supplying. This suggests there are some economies of scale to be gained from increasing the number of items supplied.

The costs in the tables above will also be able to be used to set charges for the standard interlibrary loan and document delivery fees.

Implications of the study

The National Resource Sharing Working Group will use the final results of the Study to:

develop training in best practice,
conduct seminars to discuss policy issues,
identify and promote workflows and equipment which assists best practice, and
help libraries benchmark their operations against the general results.

The report of the Study will be released in September this year. Its availability will be widely advertised, and the final results promoted through papers at conferences and seminars.

In the lead-up to the report, the data continues to be analysed to elicit other effectiveness and efficiency measures. The data will continue to be a rich source of information about ILL/DD operations in Australia, and this data will help informed debate about the future directions of ILL/DD in Australia.

A a