Learning from the BIDS IBSS Online Trial; an Exercise in International Co-operation

BIDS [1] has been providing the academic community with online access to bibliographic databases since 1991. When the BIDS ISI service was launched, it was believed to be the first of its kind - a nationally networked service providing end-user access to a large, commercial bibliographic database. Licensing for many of the services in the UK is organised by CHEST [2], the centrally funded body that negotiates special rates for software and data for the higher education sector in the UK. Both CHEST and BIDS are sponsored by the JISC (Joint Information Systems Committee) [3] which is responsible for the UK HE sector of the Internet, and a wide variety of services and projects. The BIDS service operates on the campus of the University of Bath.

Following on from the success of the ISI service, several more databases have been added to the portfolio, including The Royal Society of Chemistry databases, EMBASE, the Education databases (ERIC and BEI), INSPEC and, the subject of this paper, IBSS. With accesses to all databases now running at around 15,000 a day at peak times, the BIDS service is one of the largest of its kind, anywhere in the world.

Since 1995, BIDS has also provided access to full text material. Commencing with three publishers that were part of the UK's Pilot Site Licence Initiative [4] (Academic Press, Blackwell Science, Blackwell Publishers), the service has been steadily built up and at the time of writing had nearly 600 journal titles, with nearly 900 already committed for 1999 from 14 different publishers. Originally known as JournalsOnline, the service has recently been re-badged as ingentaJournals.

IBSS Online is the networked version of IBSS - the International Bibliography of the Social Sciences [5]. Originally set up by the International Committee for Social Science Information and Documentation (ICSSD) under the auspices of Unesco, it is now compiled by the British Library of Political and Economic Science (BLPES - the library of the London School of Economics). IBSS Online receives support from the BLPES [6], the Economic and Social Research Council (ESRC) [7] and the Joint Information Systems Committee (JISC) of the UK Higher Education Funding Councils.

The database contains references to articles and book reviews from academic social science journals, and also covers social science monographs and selected chapters from some multi-authored monographs. Over 2,300 social science journals are indexed regularly each year and over 4,000 are represented in the database as a whole. Around 30% of records refer to materials in languages other than English, and references are drawn from publications in over 100 countries around the world. Around 90,000 records are now added each year to the service. Figure 1 illustrates the geographic source of the journals which are indexed by IBSS.

The database is divided into several fields and, to assist with accurate searching, articles are categorised in a number of different ways including document type, discipline code, subject descriptors and geographical descriptors. Most records do not contain article abstracts - at the time of writing, just over 5% of journal articles currently being added to the database include abstract texts. Funding has now been approved, however, for abstracts to be included for 50% of journal articles, starting around May 1999. Foreign language titles have an English translation added to aid searching.

With records continuously collected since 1951, IBSS Online is said to be the most chronologically extensive online service in the social sciences. The complete dataset gives a picture of the development of the social sciences and the history of ideas over the last half century, as well as contemporary insights into historical events such as the Suez Crisis, decolonization in Africa and Asia, and the full course of the Cold War. As the chart shows, a large proportion of references are drawn from publications outside the United Kingdom. Coverage from Russia and Eastern Europe is particularly good for the early data.

The BIDS IBSS Online service was launched in January 1995, and use of the service in the UK has grown each year. Figure 2 shows the average number of users each day since the start of the service. The pattern of usage is similar to other services showing an annual cycle of a sharp increase each October as the university year commences, followed by a short, sharp dip around Christmas, a smaller dip around Easter, then a gradual fall-off during the last term of the year and over the summer vacation before the next sharp October increase.

In June/July 1997, a BIDS representative visited a number of universities in Australia and talked to several people about the services that BIDS could offer. Although previous knowledge of the BIDS service was limited, when the services were described and demonstrated, there appeared to be considerable interest. Service response times seemed generally quite respectable, despite the distance, and there was a lot of enthusiasm for the way in which BIDS was integrating its bibliographic databases with full text services. There was, however, some concern expressed about the (then recent) changes to the networking contracts for Australian universities that would result in higher charges for international traffic.

Nevertheless, the idea of delivering services to Australia was further explored, but a problem for BIDS was the restrictive licensing arrangements that applied to most of the centrally negotiated database deals. Products such as ISI's Citation Indexes were available to UK Higher Education establishments at very attractive prices, but the licences specifically excluded access by non-UK organisations without further negotiation with the suppliers.

The BIDS IBSS Online service was, however, subject to rather different licensing arrangements. Funding for the service came jointly from the JISC, the ESRC, and IBSS. The data, which is also available on a SilverPlatter CD-ROM, is mounted for network access exclusively on the BIDS system. Although administered as a subscription service with individual institutions required to sign licences, the licences were free to UK Higher Education. The costs of running the service in the UK were covered by grants from the JISC and the ESRC, with the IBSS unit also providing its support free of charge.

Following the BIDS visit in 1997, CAUL [8] made an enquiry about the possibility of accessing IBSS Online in Australia. Discussions took place with all the UK interested parties, and there was general enthusiasm for a positive response. IBSS were actively interested in extending use of a service that was considered to be of great interest and value to Social Scientists in many parts of the world. The ESRC which had put substantial funding into the project was also keen to see widespread use of the data. The JISC meanwhile had been forging links with the university community in Australia, and had conducted a high level visit in October 1996 exploring areas of mutual interest. Finally, BIDS was keen to extend use of its services outside the UK. Part of the motivation for this was to make the BIDS full text services more attractive for publishers by expanding its user base, thereby helping to build up a useful collection of full text material for use both within the UK and elsewhere.

Supporting services to Australia and New Zealand had particular attractions for a British-based service. Although the distance and time shift presented some potential problems, not least the issue of support, especially by telephone, it also had some very positive benefits. Both the working day and the academic year are almost exactly out of phase with the UK.

In the UK, the heaviest use of services such as BIDS takes place during the first and second terms of the academic year, ie October to December, and January to March or April. In the southern hemisphere, where academic years coincide with calendar years, peak demand is likely to be during the two semesters (March to May, July-October).

The time-shift effect is more complex to work out, since there are four time zones to take into account (three in Australia, and a fourth in New Zealand). It is further complicated by daylight saving time shifts that only apply to certain states in Australia. Nevertheless, in South-East Australia for example, in the period from late October to late March, (the period of heaviest use in the UK, and the daylight saving period in Australia), the time shift between the two countries is 11 hours, giving an almost perfect mirror-imaged day. In the UK over the 24 hours, heaviest demand is in the morning and afternoon, with peaks around 11am and 2-3pm. This is the period of lowest use from Australia and New Zealand as the graph of the trial period shows (Fig 3).

CAUL and CONZUL (Committee of New Zealand University Librarians) acted as co-ordinators for the trial and mailed out messages inviting expressions of interest. In response, all 7 universities in New Zealand and 25 Australian universities volunteered to take part. The trial commenced on 23^rd March and ran until the end of June. A full list of participants appears in Appendix 1.

The co-ordinator at each site was issued with a single, shared, username and password for the site (authentication, using the JISC supported ATHENS [9] system, was backed up with IP address checking), and BIDS posted a package of support material to each institution consisting of 25 step-by-step user guides, and 25 promotional flyers.

As mentioned earlier, a feature of BIDS bibliographic database services is the linking to full text articles where these are available online. After each search, before displaying the hits, the system tries to match each bibliographic database hit against all the article titles in the full text service (then called JournalsOnline, now re-named ingentaJournals). For every match, a hypertext linked button is shown in the display which leads to the full text article (subject to licensing arrangements).

Two major academic publishers agreed to take part in the trial - Academic Press and Blackwell Publishers. AP were already conducting an extended trial of their IDEAL service in Australia, and agreed to additionally allow access to the material via the BIDS JournalsOnline links. They also agreed to allow access by New Zealand sites. Blackwell Publishers also agreed to participate in the experiment. In total, 174 titles are available from AP and 198 from Blackwell Publishers. There were however relatively few matches between AP's titles and IBSS's coverage. There was a better correspondence between Blackwell Publishers' portfolio and IBSS. Nevertheless, the number of article matches was likely to be relatively small, as none of the full text material from these two publishers went back earlier than 1996, whereas the IBSS database coverage extends back to 1951.

Part-way through the trial, BIDS made a promotional visit to several cities in Australia and also visited New Zealand. Co-ordinators were encouraged to invite academic staff and researchers from both their own and neighbouring institutions to the presentations and demonstrations. These took place in May at Curtin University of Technology, Macquarie University, University of Adelaide, University of Melbourne, University of Sydney, ANU, Griffith University, and the University of Queensland. A presentation and live demonstration was given at each of these, and there was generally a good turnout and, in many cases, a respectable number of academics attending alongside library staff. A visit was also made to RMIT, but they had soon realised that the subject matter didn't fit well with the interests of their staff or students.

Usage varied greatly from site to site. The University of Melbourne was the most enthusiastic, with 340 sessions recorded during the trial. This compares with the least active site with only 3 accesses during the 3 months! The mean use was 75 accesses during the period of the trial. The sites that decided to take out licenses averaged 91 accesses, while those that didn't averaged 65. On the whole, New Zealand sites tested the service more energetically than Australian institutions, with an average of 92 sessions per site compared with 70 in Australia.

The only significant problem that was widely reported was difficulty in accessing the service from off-campus. The reason for this was that, as mentioned earlier, authentication was by a mixture of ATHENS-administered usernames and passwords, coupled with IP address checking. For the trial, only one username was issued per site (simultaneous access by several people using the same username is not a problem). Because this didn't offer any security, IP address checking was also invoked, using the IP addresses supplied by CAUL, CONZUL or the sites. The checking works by using a lookup table to translate the IP number of incoming calls into an IP site name, and then checking the name against the list of subscribers.

It emerged during the trial however that significant numbers of users were using dial-up (typically from home). What is more, the dial-up service provider used was often not the university itself, but a commercial company. This made a reliable IP name check either difficult or impossible, especially for those using dynamic IP address allocation. It was however reported that, in some cases, Australian Internet service providers had co-operated with universities by providing a block of IP addresses for their exclusive use. This, in theory, would allow these addresses to be added to the lookup table. Another problem was that not all sites appeared to have registered the IP addresses of all their systems, even those on campus.

In early October a follow-up questionnaire was e-mailed to all participating sites, with a slightly different questionnaire for subscribing and non-subscribing sites. University of Ballarat only gave a very limited response because of resource issues. Due to a misunderstanding, one site (Curtin) was erroneously categorised as non-subscribing, when they had in fact decided to take out a subscription. Details of the questionnaires and a summary of responses appear in Appendices 2 and 3.

The response rate was good with only 6 of the 32 participating sites failing to return the e-mailed questionnaire. All 7 of the New Zealand sites responded, and all but one of the sites which decided to take out a licence sent in a response. BIDS is grateful to those who took the time and trouble to complete this exercise.

The trial

Sites were asked who carried out the testing, and how intensively they judged the testing to have been performed. All but two of them had allowed both library staff and academic staff and students to try the service. 5 of the 9 responding subscribing sites reported intensive testing, while all but 2 of the non-subscribers said they judged the service to have been tested 'lightly'.

The Data

A particularly striking response was received to the question on the attractiveness of the time span of the data. A feature of the database is that coverage extends back to material published in 1951. 18 responders reported this as 'very attractive' and a further 6 as 'marginally attractive'. This is especially interesting in the light of the current enthusiasm for networked electronic journals, since very few of the latter currently cover more than the last 2-3 years of published material.

Less surprising, as the sites had volunteered to take part in the trial, was the finding that all the responders thought that the data was either very relevant (18) or at least marginally relevant (6).

Of more interest was the answer to the question (only posed to non-subscribers) as to how much importance they placed on the absence of abstracts in the data when making their purchasing decision. [As mentioned earlier, there are in fact a small number of records with abstracts; abstracts are currently being added for just over 5% of new records. Increased funding for IBSS will see abstracts being added to approximately 50% of journal articles from May 1999 onwards.]

Of the 17 responses, only 3 thought the absence of abstracts was unimportant. 9 thought it was a very important factor, and a further 5 thought it a marginally important factor.

The Service

Of special interest were the answers to the question about response time. Clearly the fact that the UK is about as far away as it is possible to get from Australia and New Zealand might have meant poor or at least erratic response times. In fact 12 of the sites rated the response time as 'good' and a further 11 thought it was 'acceptable'. Only one site rated it as 'poor' and one somewhere between acceptable and poor. It is possible that local networking problems might have been a factor here. The author, when demonstrating the service at a number of sites, encountered quite wide variations in response times though generally they seemed remarkably good.

Less reassuring were the answers to the question about how easy the testers found the service to use. Overall, only 6 sites rated the service 'very easy' to use, with the majority (17) regarding it as 'fairly easy', and 2 reporting it as difficult. The word 'clumsy' was used by three sites in their comments on the interface. This was echoed by responses to the question about features of the service that were not liked, with the user interface being mentioned most often.

It is worth saying here that a greatly revised user interface was introduced into the BIDS service shortly after the trial period, making use of frames and javascript to present a better looking and easier to understand set of screens. In the UK this development has been generally welcomed.

Somewhat confusingly, the critical comments on the interface were to some extent countered by the answers to the question about which features of the service were most appreciated, where 17 of the 23 sites answering this question mentioned the interface. However, if the order of responses was taken into account, the links to full text articles was a slightly more appreciated feature. The online, context-sensitive, help was given a fairly low rating.

General - Purchase Factors

The last question on the two questionnaires consisted of a list of features which were suggested as possible key factors in the decision to either purchase or not purchase. Of the sites who decided to take the service, price and data relevance were easily the most commonly quoted factors, with the interface and the full text links getting very low scores. This seems to contradict the earlier finding where the full text links were greatly appreciated. However, in practice, the full text links were fairly sparse and their usability will depend on local site subscriptions to the electronic versions of the journals.

Of those sites who didn't choose to subscribe, overwhelmingly the most important factor that was quoted was 'general budget problems'. 14 of the 16 sites mentioned this, 6 of them making it the most important factor out of the five that were offered. Also important were price and relevance of the data. Despite the negative comments about it, the service interface was only mentioned by 4 sites as a purchasing decision factor, though 3 of these quoted it as the most important reason.

A price of £1200 had been fixed for those sites that decided to commit to the service by the end of the trial, and £1500 pa for any that decided to join later. These are institutional licences, permitting unlimited access by any members (staff or students) of the institution.

There had been suggestions, particularly from CAUL and also from the CEIRC (CAUL Electronic Information Resources Committee), that a tiered pricing structure should be considered, possibly related to the number of sites who wished to take up the service. This was discussed but, given the relatively low price of the service, it was felt to be an unnecessary complication. Instead, a discount for early commitment was offered.

In March 1997 a User Survey and Evaluation of the IBSS Online Service [10] was commissioned by the BLPES on behalf of the JISC and the ESRC, and carried out by the Tavistock Institute in London.

The survey took the form of a postal questionnaire survey of academic staff, and post-graduate students, augmented by a shorter 'online' version of the questionnaire for users of the (then new) IBSS Online web-based service, together with a separate postal questionnaire for social science librarians. In addition, a number of background interviews were undertaken, and three focus groups were held with librarians, post-graduate students and academics.

Some of the findings relate closely to the service trial in Australia and New Zealand. Although researchers valued the coverage of books and journals published outside the USA, and also coverage of small and specialist journals, they said they missed the ability to search and retrieve abstracts (see earlier comments on future developments). There were also some negative comments on the interface, although at the time of the survey there was a split of users between those using the old telnet interface and those using the new web interface.

Coverage of materials going back several decades was seen as particularly useful, supporting the findings described earlier.

A particular problem in the UK was to try to bring this particular database to the attention of users who by the time of its introduction were thoroughly used to the ISI Citation Indexes, and only thought of BIDS as a service providing access to ISI.

Some Lessons

Before the trial, it was thought that the remoteness of the service might turn out to be one of the most important factors in discouraging take-up of the IBSS Online service. It was felt that the combination of slowness and unreliability of long-distance networking, and the difficulties of communicating with those providing day-to-day support would discourage many sites from considering this as a mainstream service.

In fact it transpired that these were among the least important considerations. In reality it seems that the long-distance connections of the Internet are now sufficiently fast, reliable and mature to make delivery of services such as this over very long distances a practical proposition.

It is the author's view that, while it is technically possible to set up, promote, sell and support services such as this remotely, with no personal contact, in reality putting a face to a name (or e-mail address!) makes it significantly easier when it comes to dealing with some issues, particularly if there are problems that need addressing.

One of the most consistent pieces of feedback on the data was the attractiveness of the extended time coverage of the database, with coverage of nearly half a century of publications. As mentioned earlier, this should give pause for thought among those who see an academic publishing revolution just around the corner with the advent of full text publishing online. In reality, in many disciplines, the volume of material available this way is unlikely to provide more than a fraction of the needs of most research workers for many years to come.

In the meantime, bibliographic databases such as IBSS with extended time coverage are likely to be popular complementary services. The type of integration that BIDS (ingenta) has managed to achieve, with links from bibliographic search results to full text material where available, is likely to be a popular solution.

Unsurprisingly, the availability of abstract text was a highly desired feature for many people testing the service. As mentioned earlier, the majority of articles in the IBSS database don't currently contain abstracts, and this was a significant purchasing decision factor for some sites. It may be that, for a discipline such as the social sciences, article titles alone are not necessarily a very good guide to the precise content of the article (unlike many science and technology articles that tend to have very long and detailed titles).

In summary, the main conclusions from this exercise are:

[1] BIDS - Bath Information & Data Services URL http://www.bids.ac.uk/, now part of ingenta ltd. URL http://www.ingenta.com/

[2] CHEST - the Combined Higher Education Software Team, negotiates for software, data, training materials etc for UK Higher Education URL http://www.chest.ac.uk/

[3] JISC - Joint Information Systems Committee URL http://www.jisc.ac.uk/

[4] PSLI - the Pilot Site Licence Initiative URL http://www.jisc.ac.uk/progs/index.html#pilot

[5] IBSS - The International Bibliography of the Social Sciences URL http://www.lse.ac.uk/ibss/

[6] BLPES - The British Library of Political and Economic Science URL http://www.lse.ac.uk/blpes/

[7] ESRC - the Economic and Social Research Council URL http://www.esrc.ac.uk/

[8] CAUL - Council of Australian University Librarians URL http://www.anu.edu.au/caul/

[9] ATHENS - a JISC supported system for access management URL http://www.athens.ac.uk/

The University of Adelaide

Australian Defence Force Academy

Australian National University

University of Ballarat

Bond University

Central Queensland University

Curtin University of Technology

Deakin University

Flinders University of South Australia

Griffith University

La Trobe University

Macquarie University

University of Melbourne

Monash University

University of New England

University of New South Wales

University of Newcastle

University of Queensland

Royal Melbourne Institute of Technology

University of South Australia

Swinburne University of Technology

University of Sydney

Victoria University of Technology

University of Western Sydney

University of Auckland

University of Canterbury

Lincoln University

Massey University

University of Otago

University of Waikato

Victoria University of Wellington

Between April and June this year, you took part in a trial of the BIDS IBSS Online service. Both BIDS and IBSS are very interested in finding out how the trial was received, and how your decision to purchase was reached. We would therefore be very appreciative if you could spend a few minutes answering the following 9 questions.

The Trial

Q1. Who from your institution was involved in testing the service

[a] library staff only

[b] academic staff/students only

[c] library staff and academic staff/students

a=0, b=0, c=8

Q2. Do you judge the service to have been tested

[a] intensively

[b] lightly

a=4, b=4

The Data

Q3. Did the data appear relevant to the subjects studied in your institution?

[a] very relevant

[b] marginally relevant

[c] not at all relevant

a=6, b=2, c=0

Q4. How attractive is the feature that the database extends back to 1952

[a] very attractive

[b] marginally attractive

[c] unimportant

a=6, b=2, c=0

The Service

Q5. Were service response times generally

[a] good

[b] acceptable

[c] poor

a=5, b=2, b/c=1

Q6. Overall, how easy did users find the service to use

[a] very easy

[b] fairly easy

[c] difficult

a=2, b=6, c=0

Q7. Which feature(s) of the service were most appreciated

[a] the interface

[b] the online help

[c] the links to full text articles

[In order of importance, type one or more of a, b, or c below]

a=6, b=4, c=7

[Weighted with 1^st=3, 2^nd=2, 3^rd=1 gives: a=14, b=6, c=18]

Q8. Were there any features of the service that were not liked

See text of article.

General

Q9. Which of the following were the key factors in your decision to purchase the service (if more than one, list in order of importance)

[a] price

[b] relevance of data

[c] service interface

[d] links to full text

[In order of importance, type one or more of a, b, c, or d below]

a=7, b=6, c=2, d=3

[Weighted with 1^st=4, 2^nd=3, 3^rd=2, 4^th=1 gives: a=22, b=22, c=4, d=7]

Between April and June this year, you took part in a trial of the BIDS IBSS Online service. Both BIDS and IBSS are very interested in finding out how the trial was received, and how your decision not to purchase was reached. We would therefore be very appreciative if you could spend a few minutes answering the following 10 questions.

The Trial

Q1. Who from your institution was involved in testing the service

[a] library staff only

[b] academic staff/students only

[c] library staff and academic staff/students

a=2, b=0, c=15

Q2. Do you judge the service to have been tested

[a] intensively

[b] lightly

a=2, b=13, a/b=2

The Data

Q3. Did the data appear relevant to the subjects studied in your institution?

[a] very relevant

[b] marginally relevant

[c] not at all relevant

a=12, b=4, a/b=1c=0

Q4. Although the database includes keywords fields, most records do not

include any abstract text. In your purchasing decision, how important a

factor was the absence of abstract text

[a] very important

[b] marginally important

[c] unimportant

a=9, b=5, c=3

Q5. How attractive is the feature that the database extends back to 1952

[a] very attractive

[b] marginally attractive

[c] unimportant

a=12, b=4, c=1

The Service

Q6. Were service response times generally

[a] good

[b] acceptable

[c] poor

a=7, b=9, c=1

Q7. Overall, how easy did users find the service to use

[a] very easy

[b] fairly easy

[c] difficult

a=4, b=11, c=2

Q8. Which feature(s) of the service were most appreciated

[a] the interface

[b] the online help

[c] the links to full text articles

[In order of importance, type one or more of a, b, or c below]

a=11, b=7, c=10

[Weighted with 1^st=3, 2^nd=2, 3^rd=1 gives: a=27, b=12, c=28]

Q9. Were there any features of the service that were not liked

See text of article

General

Q10. Which of the following were the key factors in your decision not to purchase the service (if more than one, list in order of importance)

[a] price

[b] relevance of data

[c] remoteness of the service

[d] service interface

[e] general budget problems

a=8, b=8, c=3, d=4, e=14