Rivers of knowledge

9th Specials, Health and Law Libraries Conference

Library-based EBH training in a public hospital

R M Sladek
Repatriation General Hospital, Adelaide, SA

Introduction

Evidence-Based Healthcare (EBH) has infiltrated nearly every facet of healthcare. In a matter of ten years or so, it has spread with an almost missionary zeal, and has been likened to a movement of holy proportions.[Grahame-Smith D 'Evidence-based medicine: challenging the orthodoxy' J R Soc Med 198; 91 (Suppl. 35):7-10] As an indexing term, Evidence Based Medicine (EBM) appeared in Medline as recently as 1997, and scanning any current health journal display will provide an array of related articles currently being published. EBH is an unavoidable issue in healthcare today.

Over the past two years I have developed and run EBH training workshops at the RGH (Repatriation General Hospital) in South Australia. The first workshops were run in early 1999. This paper will outline these training programs: their development and content. I will conclude with my views of the implications for professional development of librarians. As this paper is in part a reflection of my professional practice, it is unavoidably written in the first person, so apologies to the literary purists!

Background

My first real exposure to EBM came with attendance at an ALIA Health Libraries Professional Development Activity, run in conjunction with the Australasian Cochrane Centre in Adelaide in late 1998. During this workshop on 'EBM and the Cochrane Library', the basic tenets of EBM were outlined - what it was, and why it was important. It was suggested that participants then go back to their institutions to offer similar training there.

I considered myself capable and qualified to offer training in using the Cochrane Library, however it seemed to be fairly pointless if the contextual information about EBM was not provided. A one-day workshop was insufficient training. Hence began a process of self-education.

Basically, I did what I usually make no time for. I read. I read until I felt comfortable with the terminology. I read until I understood. I wrote a 45-50 minute introductory lecture to EBM, based on, and quoting, relevant sources. This was followed by a two-hour hands on training program using the Cochrane Library. I ran a trial of this combined three-hour workshop with two colleagues.

Trial of EBM training

I then set up a trial of four key hospital staff. The most likely to give honest and critical feedback were selected, including our principal research officer; a physiotherapist who was a long-standing member of the Research and Ethics Committee, and a pharmacist who wrote systematic reviews of literature. Their feedback was unanimously supportive of offering the training to the broader hospital community. I specifically invited detailed comments, and received constructive criticisms. For example, one participant gave me an easier way to explain an Odds Ratio. In hindsight, this was a crucial stage in the development of the training. It enabled the content to be fine-tuned, and I gained confidence in the value of the training. The content of the introductory lecture is included as Appendix A.

In addition to the Workshop, key books were purchased and a bibliography of all relevant material our library held was prepared for distribution along with other readings for participants. During this developmental stage ABC TV ran a series of programs called 'Too Much Medicine'. Part three of this series, 'Too Many Tests' was a very general introduction to the concept of Evidence Based Medicine.

This video was purchased and offered over several lunchtime viewings. The rationale was that it would firstly serve as an easy viewing, enjoyable introduction to the topic, and secondly, it might address casual interest in the topic so that formal workshops would attract only those who were really interested in the next level of understanding.

The training was advertised via our networked e-mail system on two occasions, and once in our staff newsletter. The response was extremely encouraging. In a four-month period approximately 100 participants attended the workshop based on the initial promotion and word-of-mouth recommendation. Training positions in the workshops I had scheduled quickly filled, and additional sessions were held. Both the chief executive officer and medical director attended training, and on the recommendation of the latter, the Chairman of the Board attended a session. Allied Health, Nursing, Medical and Research staff categories had the highest number of participants.

One of the byproducts of commencing this training, was my increasing awareness of how much there was to learn. In part this was managed by inviting questions and providing frank admission of the gaps in my knowledge. Every session was then followed up with phonecalls to local experts, including Phillipa Middleton at the Australasian Cochrane Centre, to answer specific questions. Eventually all questions and answers were posted on our computer network for everyone to read. In this way I facilitated others in learning about EBM, as I learned more.

Overseas study

After such encouraging feedback, I began to wonder where I could further my knowledge. I discovered a program called the 'Critical Appraisal Skills program' (CASP) in the United Kingdom. This program is funded through the NHS (UK National Health Service) to help decisions makers and those who influence them to develop skills in critical appraisal. Based at the Institute of Health Sciences in Oxford, CASP aims to promote the delivery of evidence based healthcare. They use a 'cascading model' of training where those who are trained become the next trainers.

Thanks to a grant from the Rights of Private Practice (a medical staff trust fund operating in the Repatriation General Hospital), some hospital funding and personal funds, I attended the First International 'Train the Trainer' Workshop in Critical Appraisal Skills in London, July 1999. This training provided a structure for running workshops in Critical Appraisal Skills.

This week caused a significant leap in my learning curve. I was exposed to multidisciplinary professionals from Europe, and an environment where EBM had impacted significantly in all areas of the NHS. Real attempts were being made to provide relevant training to all health care workers, and the resultant atmosphere was very conducive to learning. I believe this was a turning point in my career, because I began to really understand the broader context in which health libraries, and librarians in particular, operate.

I had undertaken that on return I would impart any relevant knowledge to my hospital community. This was a challenge! I will return to this point later, but suffice to say that the UK approach of 'cascading training' may not be the best model in Australia, and it could be well argued that there are key credentialing issues that need to be addressed.

New CASP workshop

To ascertain if a CASP Workshop was useful, appropriate or desirable, the same trial methodology was employed. Volunteers were invited through our e-mail network to attend a developmental workshop on critically appraising a Systematic Review of Randomised Controlled Trials on 'St Johns Wort for Depression'. I had too many volunteers for my trial, however this served to inform me of the level of interest in these type of workshops. I selected a varied group of ten people, and included those who I thought would be rigorous in their critique.

At the conclusion of the workshop I added a 15-minute review period, when the quality, usefulness and relevance of the workshop was discussed. After obtaining written evaluations, I discussed my role offering this training with key participants. Feedback was unanimously enthusiastic about the relevance and quality of the workshop. I then offered this CASP Workshop to the broader staff community. This workshop was run periodically until early this year, and approximately 200 have now attended. They have all included evaluation by participants, which has continued to be extremely positive.

I would like to stress again, the value of the 'trial' methodology. Because it was a new area of professional practice, this provided me many benefits. Firstly, I was open with staff about any concerns I had, and this enabled them to be specifically addressed. Secondly, I felt more comfortable running latter training having had Workshops initially reviewed by experienced, multi-disciplinary staff. Thirdly, because I earnestly sought 'real' feedback, I was able to improve the workshop.

Although in this paper it is impossible to document all aspects of this workshop, Appendix B provides a brief outline. Essentially, the same introductory lecture as for the 'EBM and the Cochrane Library' Workshop is included (unless participants have already attended same). This is followed by a twenty-minute introduction to key statistical and research concepts, before embarking on the actual appraisal. This comprises a CASP checklist of key questions with the format of the discussion modeled largely on the training I received in the UK. The onus is on the group to discuss the questions, rather than the facilitator. There may be awkward moments when this begins, but usually participants take responsibility for reading and appraising the paper, providing I am strong enough not to interrupt that initial silence!

I am developing a new workshop specifically for clinicians, focussing on how to find answers to clinical questions. This is a new challenge, as the model for how to best search for the evidence in clinical practice, preferentially using sources other than Medline as the starting point, is constantly evolving as new sources are developed.

Benefits

The involvement in EBH training has led to many benefits for the hospital, the library and library staff, and the profession.

For the hospital...

Delivery of the training has provided the hospital with staff development opportunities for clinical and research staff in particular. It has utilised internal resources to provide training on-site, at convenient times for staff. Staff could be summoned for emergencies - something important for clinicians - and training was tailored to specific local needs. For example, the hands-on tour of the Cochrane Library was modified readily with a different subject focus depending on the background of those attending. The training also provided opportunities not available elsewhere at the time, and in a multi-disciplinary setting. This meant that individuals who had worked at the hospital for years in their own section were often introduced to others who were similarly long-standing staff.

For the library and its staff...

The library and its staff have benefited in various ways. It has provided an opportunity for promotion of services and staff. Participants could be referred directly to resources, and indirect publicity was achieved by non-users reading e-mail messages about the library's activities, and word of mouth. This exposure is invaluable in terms of being seen to be clinically relevant by users and non-users alike.

We now have a better knowledge of resources to support EBH, enabling better selection and understanding of sources to support reference work etc. This has raised confidence in our clinical relevance. We now know more staff by name across the hospital, and perhaps more importantly, more staff know us by name. We have an increased professional profile.

We now have a Library Trust Fund. Whilst CASP Workshops cannot be run for profit, non-hospital staff have been able to attend both the 'EBM and the Cochrane Library' and 'CASP' Workshops for a small cost recovery charge. In addition, workshops outside of the hospital have resulted in income. Whilst only small amounts, this is a useful fund for the library.

For me...

Personally, developing and running the EBH workshops has provided me with the opportunity to develop, in terms of knowledge, skills and attitudes. For example, I have an improved understanding of clinical issues, more experience communicating with different professionals, and have had a steep learning curve in working as a facilitator and educator. My IT and writing skills have been improved.

I have had the opportunity of overseas travel, been invited to present workshops outside of the RGH, and provided several series of lectures for university courses. I have gained the impetus to undertake further academic studies. My work with EBH supported my application for paid study leave to undertake the Masters. I have had a renewed enthusiasm for my professional relevance.

For the profession...

After attending the CASP training in the United Kingdom, I provided a training session to South Australian Health Librarians. Since then, I have provided training to colleagues through attendance on-site at the hospital-based Workshops. Where possible, if interstate colleagues have contacted me, I have tried to schedule Workshops to co-incide with their visits to South Australia. We need to be committed to developing each other's skills base.

These workshops have gained some profile locally, and recently I was asked to run a tailored workshop at the inaugural Australian Centre for Evidence Based Clinical Practice's first EBP Workshop. Colleagues from the Flinders University Library collaborated with me to run small group sessions after a plenary session on 'Finding the Evidence'. This type of exposure amongst clinicians and academics is positive for our profession. The cross-campus collaboration, was a positive experience for those librarians involved.

The cost

But of course there have been less favourable impacts of involvement in this training. The major issue has been workload. Training is very human-resource demanding, and it impacts both on the trainer and the staff remaining in the library dealing with absences. Library staff have been exceptionally supportive of the training commitment - these workshops would have been difficult without a genuine team approach to developing library services. Normal workload can suffer, and this can be stressful. Training can be tiring and demanding. I quickly learned not to schedule two workshops on the same day!

Colleagues may wonder how an increased workload was managed. There are two responses. Firstly, other work became less of a priority, for example, collection development. I undertook less formal library planning, but library development became a major outcome of the workshops anyway. Secondly, I believe library staff all worked harder. You can't make something out of nothing. However success is a positive reinforcer, and the rewards have by far outweighed this investment. Workshops are not run continuously, so there are recovery phases. The administrative preparation work for workshops diminishes as documentation etc is developed and re-used. Again I would stress the importance of a good team of library staff, working together. Flexibility and commitment are essential.

Professional development issues

The issues for professional development for our librarianship fall into two main areas: what is required to continue to provide high quality services professionally, and expanding our role into teaching critical appraisal skills.

The minimum requirements professionally for health librarians working with the clinical and research community include:

• Knowledge of what EBM is: its origins, strengths and weaknesses
• Knowledge of how librarians fit into the 5 stage process of EBM (as teachers/doers of Stages 1 and 2, and at least as a doer of Stage 3)
• Understanding of new sources of information: their origins, development and why they are important (secondary journals; systematic reviews)
• Understanding of research methodology
  • Causal relationships in health
  • Key research designs and relative merits/methodological bias
  • Systematic Reviews
  • Key statistical concepts
• Publication/Indexing Issues
  • Research process/scholarly publishing
  • Publication Bias
  • Indexing Bias in key resources
• What constitutes good evidence?
  • Levels of Evidence
• Understanding the Cochrane Collaboration and Cochrane Library
• Library issues
  • Relative merits of different sources for levels of evidence
  • Knowledge of key sources/resources to support EBH
  • Evidence searching filters
  • Staying on top of emerging issues
  • Communicating about clinical issues/clinical knowledge

Credentials

The final issue for comment is that of credentials. The professional world operates with credentials underpinning all areas of practice, and healthcare is notably one of these. I suspect that in my case, my relative ease of public speaking and group work, matched with concurrent studies in research methodology as a Psychology Honours pre-requisite, assisted me. These are personal qualifications (and/or skills), and not necessarily shared by others. Our profession needs to consider the credentials appropriate for involvement in EBH training.

Future of CASP Training

The challenges led me to commence a Masters in Public Health. This has been relevant in its coverage of biostatistics and epidemiology, and research methodology. I can certainly recommend this curriculum for its relevance to health librarians. I consider such credentials, or similar, necessary for an educator role in critical appraisal. I have suspended running the CASP Workshops mainly because there is no infrastructure in Australia to support cascading training. In the UK there are some tremendously skilled public health clinicians and academics behind the program. I liken my CASP experience at my hospital to a 'grass roots level' response to a need and interest, but that now requires formal structure if it is to continue. In the absence of any government initiatives similar to those in the UK, then it must come from rigorous academic study.

Conclusion

The positive experience of developing and providing EBM and CASP workshops at the RGH clearly demonstrates the potential role librarians have in the emerging paradigm of EBH. As a profession, we need to examine, prepare and embrace this role. For some librarians, this may mean attaining a core set of minimum competencies and we should commit ourselves to pursuing the best mechanisms for effecting this learning. For others, it may mean seeking out further relevant qualifications such as a Masters in Public Health. The important message is that continuing professional development is now a crucial issue for health librarians.

Appendix A

'Introduction to EBM' workshop

Structure of the Workshop:

Part 1: Introductory lecture

Part 2: Hands-on tour of the Cochrane Library

Part 1: Content

What is EBM?

Terminology

Importance of EBH - why bother?

Problems of narrative reviews

New Sources of information

State of Medical Information

Indexing and publication Bias

RCTs

Levels of Evidence

Cochrane Systematic Reviews

Comparison of Cochrane Library vs Medline

Cochrane Logo

Odds Ratio Diagram

EBH Issues/Controversies

Conclusion

Appendix B

'CASP Workshop'

Structure:

Part 1: Introductory Lecture (as for 'EBM and the Cochrane Library')

Part 2: Introduction to Critical Appraisal

Part 3: The Appraisal using the CASP Checklist

Part 2: Content

General:

• Origins of the Workshop and how it was developed.
• Goals: not to comprehensively teach critical appraisal skills, rather to (1) learn how to use a CASP tool to critically appraise a Systematic Review, and (2) actually appraise a Systematic Review on 'St Johns Wort for Depression'.
• Review of EBM: a 5 stage process of converting information into an answerable question, finding, appraising and applying evidence, evaluating performance.
• Background of CASP in the UK...helping decision makers/ those who influence them to develop skills in critical appraisal and to promote the delivery EBH.
• Why does good evidence from research fail to get into practice?

Specific:

• Critical Appraisal - what is it?
• 3 key questions: how trustworthy is the study (ie, its validity)? What is it telling us (ie its results)? How useful is it (ie its relevance)?
• Systematic Reviews and the need for critical appraisal. Poorly done reviews will simply compound the flaws of the individual studies it includes.
• Introduction to the CASP Tool: outline of questions and how it was developed

Research design:

• 5 critical points: randomisation, blinding, double blinding, all participants are followed up, intention to treat analysis
• Review of design features of an RCT and discussion of critical points

Statistics:

[Participants encouraged not to be phased by statistics - emphasised they are tools and understanding the concept is the most important aspect]

• 5 critical points: measures of risk (odds ratio; relative risk), plotting risk on a graph, confidence intervals, p values

CASP Tool:

The group is facilitated in working through the CASP Checklist, essentially as suggested by CASP in the UK. Session is concluded with invitations for questions and distribution of more recent reviews on the topic.