STRAIT to the future

8th Asia-Pacific Specials, Health and Law Librarians Conference

Comparing the value and impact of information obtained by health professionals undertaking literature searches with those undertaken by librarians

Patricia Scolaro
King Edward Memorial and Princess Margaret Hospitals, Perth, WA

Keywords: Literature searching; end users; networked databases

Abstract

A study was undertaken in 1993 to examine the impact and value of information obtained from literature searches undertaken by library staff, and from documents supplied by the library staff at Princess Margaret Hospital for Children. This study was repeated at King Edward Memorial Hospital for Women in 1996.

The quality of the information provided was evaluated for relevance, accuracy, currency, and the degree of satisfaction with the service. The cognitive value of the information provided was assessed according to the level of new information provided, the time spent reading the information and the number of colleagues who saw the information. Some indications of the way the information provided by the Library added to knowledge were obtained.

The impact of the information on patient care, research, teaching, publication, case presentation or management was assessed on a scale of 1 to 4. The study showed that the information provided by the Library had a significant impact on patient care, research, teaching, and publication. Some indications of the specific nature of the impact of the information provided by the Library were obtained.

A further study was undertaken in 1999 at King Edward Memorial and Princess Margaret Hospitals to assess the value and impact of information obtained from literature searches undertaken by health professionals using networked databases and information sources. This paper compares the outcomes of end user searching by health professionals with those undertaken by library staff and discusses the implications for libraries and health professionals.

Introduction

The availability of the key databases and information sources in medicine, nursing and allied health in electronic format has revolutionised the way information is retrieved in health care. Most health professionals now have access to Medline free of charge through services such as Pubmed, Biomednet and Internet Grateful Med. Hospital libraries have been quick to take up the CD-ROM version of Medline, CINAHL (Cumulative Index to Nursing and Allied Health) and many are now networking these databases through their hospital networks.

Health professionals have always undertaken literature searches using published indexing and abstracting services. The time consuming nature of such searching meant that the preparation of bibliographies became the role of the hospital librarian. The advent of on-line databases meant that health professionals were able to obtain a printout of references, including abstracts. This coupled with the fact that on-line searching required specialised training, and that the databases were fairly structured, meant that literature searching continued to be the role of the hospital librarian. The development of user friendly CD-ROM versions of databases which allowed key word searching and which were supported by intelligent searching software has seen a shift from librarian mediated searches to end user searching.

Several concerns have been expressed about this both by health professionals and librarians. Health professionals are concerned that they do not have time to undertake searches themselves or that they are not sufficiently familiar with the databases to perform a thorough search. Concern has also been expressed that there is a temptation for health professionals to gain their information from the abstract rather than from the article. Librarians are concerned that health professionals are missing relevant information and thus diminishing their search effectiveness. Librarians are also concerned that they have lost control of one of their most powerful services and that this will change their role.

In an attempt to assess the effect, if any, of end user searching upon the value and impact of the information obtained from literature searches a survey of literature searches undertaken by health professionals was carried out at King Edward Memorial and Princess Margaret Hospitals in 1999. This survey was compared with a similar survey of literature searches undertaken by librarians at Princess Margaret Hospital in 1993 and King Edward Memorial Hospital in 1996.

Methodology

The Questionnaire (Appendix) was sent to all health professionals who were registered users of the networked databases Medline, CINAHL (Cumulative Index to Nursing and Allied Health) and HealthSTAR. Health professionals were asked to complete the questionnaire with reference to one occasion only when they had used the networked databases to search for information. They were asked to return the questionnaires to the library within two weeks. All information was treated as confidential. A follow up memo was sent to all registered users three weeks later.

The response rate of 16% was disappointing. The response rate for the earlier surveys of health professionals who had requested literature searches from the library in 1993 and 1996 was 36%. The lower response rate could be due to the fact that a proportion health professionals who register for access to the networked databases do not use them, or use them very infrequently. Some respondents indicated that they found the databases difficult to use. In the studies undertaken by King (1987) and Marshall (1992) the response rate was 57% and 46% respectively. Their higher response rate could be due to the fact that only staff who had agreed to participate in the study were surveyed.

Use of the databases by the different professional groups

There is considerable variation both in the rate of requesting literature searches and end user searching between the different professional groups. Medical, nursing and allied health staff requested literature searches from the library in the ratio 50%:15%:30% in the studies undertaken at both hospitals in 1994 and 1996. Medical, nursing and allied health staff undertook literature searches using the networked databases in the ratio 39%:23%:27% in the survey undertaken at both hospitals in 1999. Medical staff are the most frequent users of the literature, followed by allied health and nursing staff.

That nurses use hospital libraries less than other professional groups has been shown in other studies (King, 1987 and Bunyan and Lutz, 1991). Blythe and Royale (1993) postulated that nurses' rigid routines and the limited extent to which information seeking is encouraged or valued, act as barriers to the use of literature searching. At both hospitals we have made a concerted effort to encourage nurses to register for access to the networked databases and to undertake training in their use. Nurses are introduced to the networked databases at their orientation and are given every encouragement to use the databases when they visit the library. Royle and others (1995) found that nurses who received training in the use of electronic databases would continue to use them. The increasing emphasis on the practice of evidence based health care in both hospitals is also promoting the use of the literature by all professional groups.

Purpose of the literature search

Many staff indicated more than one purpose for their search. Overall the purpose of the search is indicated in the table below:

Relevance, accuracy, and currency of the information obtained

The rate of literature searches undertaken by library staff rated moderately or highly relevant, accurate, and current (91%:93%:93%) was comparable to those undertaken by health professionals (94%:98%:98%). In the King (1987) study the survey group rated the searches 88%:99%:99% for relevance, accuracy, and currency.

However, a higher percentage of searches undertaken by library staff were judged to be highly relevant, accurate, and current (73%:80%:84%), compared with those undertaken by health professionals (53%:46%:62%). The higher rate of highly relevant, accurate, and current searches reflects the higher precision rate achieved by librarians in the study undertaken by Haynes and others (1990) who showed that librarians achieved significantly better precision (58%:49%:38%) when compared with experienced and novice clinicians.

Assessing the value of information obtained from literature searches

Several indicators were used to assess the value of the information obtained from literature searches. These were, the time spent reading the information, the number of colleagues who saw the information, whether or not the information obtained reinforced existing knowledge, and the amount of new information obtained.

The results are comparable with other studies. King (1987) and Marshall (1992) reported that 84% and 79% respectively, of the physicians indicated that the information supplied by the library substantiated prior knowledge. Greenburgh and others (1978) found that clinicians shared the results of their searches with an average of 3.3 colleagues.

8% more searches undertaken by library staff supplied all new information. This can be an important factor when searching for information. Erikson and Walker (1998) point out that in clinical areas where the literature contains reports that come to different conclusions, an inexperienced searcher may not retrieve sufficient articles to realise this. The large number of colleagues who saw the information obtained from the literature searches undertaken by health professionals was due in part to a couple of searches being undertaken for case presentations where the information was passed onto a large number of colleagues. The 5% trimmed mean was 4.1.

Ways literature searches added to the knowledge of health professionals

Health professionals were asked to outline the way the literature search added to their knowledge. 10 ways in which literature searches added to health professionals knowledge were identified and the most frequent are listed in the table below.

Undertaking a literature search increases the knowledge base of health professionals principally by informing them of new research, new treatment methods, updating or extending their knowledge on a topic, and providing a review of the topic. There were differences between the searches undertaken by library staff and those undertaken by health professionals. The results could indicate that librarians are more likely to retrieve information on new topics than health professionals. Haynes and others (1990) showed that librarians achieved significantly better precision when undertaking literature searches (58%:49%) when compared with experienced physicians.

Assessing the impact of information obtained from literature searches

Once health professionals have added to their knowledge it is important that this knowledge is put into practice. Health professionals were asked to rate the impact of the information they obtained on, patient care, research, publication, case presentation, teaching, and management (administration). The percentage of those surveyed who indicated that the information had a moderate or major impact are shown in the table below.

Health professionals indicated that the knowledge they obtained from the literature searches had an impact on the activities of patient care, research, publication, case presentation, teaching, and management. The greatest impact of searches undertaken by health professionals is on patient care while the greatest impact of searches undertaken by library staff is fairly evenly spread between patient care, research and teaching. A notable benefit of all searches is that while a search may be undertaken for a particular purpose, the knowledge obtained impacts on other areas. For example 34% of searches were undertaken for the purpose of patient care and yet health professionals indicated that 81% of the searches impacted on patient care to some degree.

Best evidence and the Cochrane Database of Systematic Reviews

The Ovid version of Best Evidence and the Cochrane Database of Systematic Reviews were networked through the hospital in December 1998. At the time of the survey these two databases had been networked for 3 months, although the Cochrane Library had been available on a stand alone computer in both libraries for several years. All registered users of the networked databases had received notification of the two new databases that had encouraged their use, and pointed out the ease with which searches conducted in Medline or CINAHL could be repeated in Best Evidence and the Cochrane Database of Systematic Reviews.

Overall 10% of health professionals repeated their searches in Best Evidence and 24% repeated them in the Cochrane Database of Systematic Reviews. There was significant difference in the rates between the two hospitals for the Cochrane Database of Systematic Reviews. 9% of health professionals at Princess Margaret Hospital for Children and 39% at King Edward Memorial Hospital for Women repeated their searches in Cochrane.

This result is probably in line with what can be expected given the present coverage of the Cochrane Database of Systematic Reviews. Young and Ward (1999), and Silagy and others (1999), recognise that to date, the coverage of health care interventions provided by the Cochrane Database of Systematic Reviews is patchy. While good progress in this area has been made with over 500 systematic reviews being completed in a short time coverage will be limited for several years. There is a difference in the rate of the use of the different specialties within medicine. Young and Ward (1999) in their study of general practitioners showed that only 4% of general practitioners had ever used the Cochrane Library. Jordens and others (1998) reported that 72% of neonatologists and 44% of obstetricians surveyed had reported using systematic reviews. The lower rate at Princess Margaret Hospital reflects the narrower coverage of the Cochrane Database of Systematic Reviews in paediatrics, while the higher usage at King Edward Memorial Hospital reflects the wider coverage in obstetrics, gynaecology and neonatology.

Evaluating the service provided by the library to support the use of networked databases

When a health care worker registers to use the networked databases they are sent a file containing a set of instructions and are offered training on the use of the networked databases. Where possible, the file of instructions is delivered directly to the health care worker. The list of registered users is kept in a mail merge file so letters are addressed to each registered health care worker personally. As we are aware that the files containing instructions are liable to go astray, follow up memos have been sent approximately every six months asking staff if they require a set of instructions, or if they would like training on the use of the databases. Use of the mail merge system allows us to keep all of those registered informed of major changes to the networked databases eg. the introduction of Best Evidence and the Cochrane Database of Systematic Reviews. Memos are also sent to all Heads of Department for circulation to all staff. New medical, nursing and allied health staff are informed about the availability of the networked databases at orientation.

To evaluate the services provided by the library to support the use of the networked databases health professionals were asked if they had received a set of instructions for the use of the databases and whether or not they had received training in the use of the databases. Health professionals were asked to evaluate the instructions and the training on a scale of 1 to 4. 83% of health professionals indicated they had received a set of instructions and 51% indicated that they had received training. In the study undertaken by Haynes and others (1990) all participants were offered training but 62% indicated in their responses that they had received training. 82% of health professionals indicated that they found the instructions moderately or highly useful and 82% also indicated that they found the training moderately or highly useful. The need for training to improve searching skills was demonstrated in the study by Haynes and others (1990) where novice clinicians performed less well than experienced clinicians with respect to precision (38%:48%) and recall (27%:48%). A later study by Haynes and others (1993) showed that physicians could improve their searches to the level of experienced searchers by the fourth session.

Obtaining articles located in a literature search

Concern has also been expressed that there is a temptation for health professionals to gain their information from the abstract rather than from the article. To investigate this health professionals were asked if they had obtained journal articles from the library as a result of their search and if so what percentage were available in the library. Health professionals were asked to indicate how many articles they had requested on inter-library loan and what percentage arrived in time for their current need. 87% of health professionals indicated that they had obtained journal articles from the library as a result of their search and 65% indicated that they had requested an inter-library loan as a result of their search.

The percentage of articles available in the library is listed in the table below:

The percentage of articles requested on inter-library loan that arrived in time for the current need is listed in the table below

Only 10% of health professionals found 75% or more of the articles they wanted to read in the library. Of the 65% who then request articles on inter-library loan 56% received 75% or more of the articles in time for the current need. Libraries are continually being asked to do more with less and this has resulted in a steady reduction in the number of journal subscriptions. This has reduced the probability that health professionals will find the information they need in their own library. Document delivery systems supplement the journal collection and at both hospitals, articles required for patient care are fast tracked in order to obtain them within 24 hours. However, health professionals certainly experience barriers to finding the information they have located from a literature search in time for their current need. No information was obtained about the 13% of health professionals who did not look for journal articles following their searches.

Barriers to the use of databases

While our study did not examine why health professionals do not use electronic databases, studies done elsewhere have elucidated some of the reasons. Marshall (1990) found that a third of the users of Grateful Med surveyed 3 years later had given up searching. The reasons they stated for this included:

• system too difficult to use
• content poor or inappropriate
• system too slow
• user too busy to search

In our study those surveyed were invited to make comments about the networked databases. Several staff indicted they did not have time to undertake searches and others indicted that they found the system difficult to use.

Additional barriers to obtaining the information required identified in our study were that, the library may not have the articles identified in the search as relevant, and that not all articles requested by document delivery arrive in time for the current need.

Functionality of the databases

The ease of use of a system effects whether or not the system is used and the frequency of use. The client version of Ovid was networked in 1996 and the web version mid 1998. Both systems were run concurrently until the annual update in March 1999 and during this period users were encouraged to move to the web version. Anecdotal evidence indicates that users prefer the client version and I have written to Ovid indicating the major problems our users are experiencing with the web version. It is important that database vendors and users work together to improve the functionality of the databases as experience has shown users will cease to use systems they find difficult.

How effectively do health professionals use databases

Hersh and Hickman (1998) searched all sources of medical informatics and information science literature published between 1966 and 1998 for evaluation studies, with particular emphasis on those evaluating use by physicians. They concluded that studies have not shown that advanced searching is more effective than text word searching and that most searches retrieve less than half the given articles on a topic. Haynes and others (1990) showed that there was little difference in searching between experienced physicians and librarians. Librarians achieved significantly better precision (58%:49%) when compared with experienced physicians, but there was comparable recall (49%:48%). However, novice clinicians performed less well with 38% precision and 27% recall. A later study by Haynes and others (1993) showed that physicians could improve their searches to the level of experienced searchers by the fourth session.

Best practice for librarians

The results of this study would indicate that best practice for librarians in supporting health professionals searching networked databases includes the following:

• Implement training programmes and provide documentation and support to assist health professionals to become proficient searchers
• Inform health professionals that they are unlikely to become proficient searchers without training.
• Promote themselves as consultants to end users with both initial and difficult searches.
• Emphasise that health professionals without searching skills should have their searches done by a librarian.
• Emphasise that health professionals should seek assistance with searching until they become proficient.
• Monitor the progress of the Cochrane Database of Systematic Reviews and promote its use as the coverage increases.
• Work with database vendors to ensure the efficient retrieval of high quality information
• Monitor journal collections to ensure the best fit between user needs and the collection.
• Ensure that document delivery times are such that they ensure delivery of documents in time for the current need.

Best practice for health professionals

Health professionals who undertake their own searches need to ensure that their precision and recall are sufficient to make decisions based on the information obtained. The results of this study would indicate that best practice for health professionals undertaking literature searches on networked databases includes:

• Undertake training in the use of databases to ensure they become proficient searchers.
• Have their searches done by a librarian until they become proficient.
• Seeking assistance with initial and difficult searches.
• Repeat searches in the Cochrane Database of Systematic Reviews as the coverage increases

Conclusion

The availability of the key databases and information sources in medicine, nursing and allied health in electronic format has revolutionised the way information is retrieved in health care. The study indicated that health professionals tend to retrieve a lower percentage of highly relevant, accurate and current information when compared with librarians. Health professionals also retrieved less new information compared with librarians. From their searching, health professionals retrieve information which adds to their knowledge base and they apply this knowledge to patient care, research, publication, case presentation, teaching and management. Searches undertaken by health professionals had the greatest impact on patient care.

Best practice in retrieving high quality information in the future will depend on cooperation between librarians, health professionals, and vendors. Vendors will need to improve their products to ensure the efficient retrieval of high quality information. Health professionals will need to undertake training and seek assistance in the use of databases and information sources until they become proficient searchers. The librarian's role will be to provide training in the use of electronic databases and information sources, to undertake searches as required, and to act as a consultant for difficult or initial searches.

Author

Patricia Scolaro is Head of Department Library at King Edward Memorial and Princess Margaret Hospitals. Patricia has worked in health libraries for 9 years and previously worked in state and academic libraries and tutored in the Library School Curtin, WA. Patricia's particular interests are information management and user education.

Bibliography

Blythe, J. and Royle, J.A. 1993.
Assessing nurses' information needs in the work environment. Bulletin of the Medical Library Association 81(4):433-435.

Bunyan, L.E. and Lutz, E.M. 1991.
Marketing the hospital library to nurses. Bulletin of the Medical Library Association 79(2):223-225.

Erickson, S. and Walker, E.R. 1998.
The impact of an individual tutorial session on MEDLINE use amongst obstetrics and gynaecology residents in an academic training programme: a randomised trial. Medical Education 32:269-73.

Haynes, R., McKibbon, K., Walker, C., Ryan, N., Fitzgerald, D., and Ramsden, M. 1990.
Online access to MEDLINE in clinical settings. Annals of Internal Medicine 112:78-84

Haynes, R. B., Johnston, M.E., McKibbon, K., Walker, C.J., and William, A.R. 1993.
A program to enhance clinical use of MEDLINE: a randomized controlled trial. Online Journal of Current Clinical Trials [serial online] May 11, document 56.

Hersh, W. R. and Hickman, D.H.1998.
How well do physicians use electronic information retrieval systems: a framework for investigation and systematic review. JAMA 280(15):1347-1352

King, D.N. 1987.
The contribution of hospital library information services to clinical care. Bulletin of the Medical Library Association 75:291-296

Jordens, C.F.C., Hawe, P., Irwig, L.M., Henderson-Smart,D.J., Ryan, M., Donoghue, D.A., Gabb,R.G. and Fraser, I.S. 1998.
Use of systematic reviews of randomised trials by Australian neonatologists and obstetricians. Medical Journal of Australia 168(3):267-270.

Marshall, J. 1990.
The continuation of end-user on line searching by health professionals: preliminary survey results. In: Proceedings of the Medical Library Association Annual Meeting; May 18-24, Detroit, Michigan.

Marshall, J.G. 1992.
The impact of the Hospital Library on Clinical Decision Making; The Rochester Study. Bulletin of the Medical Library Association 80(2):161-191

Royle, J.A., Blythe, J., Potvin, C., Oolup, P. and Chan, I.M. 1995.
Literature search and retrieval in the workplace. Computers in Nursing 13(1):25-31.

Silagy, C.A., Weller, D.P., Middleton, P.F. and Doust, J.A. 1999.
General practitioners use of evidence databases [letter]. Medical Journal of Australia 170(4):393.

Young, J.M. and Ward, J.E. 1999.
General Practitioners' use of evidence databases. Medical Journal of Australia 170(1):56-8.

Appendix