Conference papers

Information literacy strategies

What a Web We Weave: Evaluating the Flexible Delivery of Information Literacy Education

Heather Drew (USQ), Wendy Abbott (GU) and Debbie Orr(CQU)

Abstract

In recent years, there has been an enthusiastic rush to exploit the unique capabilities of the WWW as an instructional medium. Librarians have been in the vanguard of those providing flexible delivery of learning materials and there are now many examples of web-based information skills programs illustrating a mix of instructional and media attributes. This paper will focus on web-based information literacy programs from various Australian universities in order to assess the effectiveness of this medium in delivering information literacy education. It will draw on assessments from students, academics and librarians, and evaluate the programs against published standards of best practice.

Introduction

An increasing number of Australian universities have recognized the need to give students the opportunity to learn a range of generic skills, including information literacy. 'Information literacy is the capacity to examine high level problem solving processes requiring a wide knowledge of information sources, systems and technologies, as well as information retrieval, management and evaluation strategies' (Bruce and Candy, 1994). There are numerous ways librarians and academics can create and deliver information literacy programs, as well as numerous opportunities for students to learn and practice the necessary skills and concepts, and the web has diversified the opportunities for everyone. The challenge lies in deciding when, how, for which client groups, and for which types of content, web-based delivery is suitable.

In order to explore the potential and appropriateness of the web for the delivery of information literacy instruction, the following programs were reviewed: The Library Research Tutorial: An Information Literacy Skills Program (abbreviated to LRT) (Griffith University); Pilot: Your Information Navigator (Queensland University of Technology); and eGO (University of Southern Queensland). A brief description of each program can be found in appendix 1.

Two main approaches to evaluating the programs were used. Firstly, the Association of College and Research Libraries (ACRL) information literacy standards were used to assess the content of the programs. The second approach was to evaluate the programs against quality criteria for the design of web-based instruction. In addition, students, librarians and academics were involved in surveying the Griffith University tutorial. In the present study, however, no attempt was made to measure the learning outcomes for students undertaking a web-based program.

From the assessment of the packages it was possible to determine a framework or methodology which librarians can use to review their individual needs before selecting, writing or using a web-based program.

ACRL Standards and the Content

The Association of College and Research Libraries (ACRL) has developed information literacy competency standards for higher education, in an endeavour to make explicit the actions required for gathering, analysing, and using information. Like all behavioural standards, students are expected to demonstrate all of the competencies described in the document, but not everyone will demonstrate them to the same level of proficiency or at the same speed (ACRL, 2000).

The second of five ACRL standards states 'the information literate student accesses needed information effectively and efficiently'(ACRL, 2000). Since the objectives behind the programs include the development of searching skills it was decided to assess how successfully the ACRL outcomes were embedded in the three reviewed programs. appendix 2 lists the ACRL outcomes and shows whether or not they could be identified in the programs, along with whether or not the skill was made explicit.

Results show that the majority of skills needed to access information are addressed explicitly in the three programs. However, the broader concepts associated with those skills were sometimes dealt with implicitly. In developing web-based programs it is important to consider the amount of descriptive detail to be included. A short overview approach ensures the user quickly learns how to retrieve an item, without devoting huge amounts of time to understanding the intricacies of the process. But there is a risk that students will not have understood sufficiently to develop the skill or gain the knowledge. A program which goes into more detail ensures the user understands the broader concepts and is therefore more likely to be able to transfer what he/she has learned to new contexts. However, such a program also bears the risk of losing the attention of the student.

Although the process was not repeated for ACRL standards 1, 3, 4 and 5 it is felt these outcomes would not have been quite so evident in the programs reviewed. For example, standard 3 states 'the information literate student evaluates information and its sources critically and incorporates selected information into his or her knowledge base and value system' (ACRL, 2000). While critical and conceptual thinking along with synthesis and processing of ideas are just as important as the mechanics of finding information, most information literacy programs to date have been centred on the outcomes outlined in the second standard. Librarians are starting to recognize the potential of using web-based programs to develop conceptual skills. For example, in 2001 staff at QUT will further develop Pilot in order to offer students the opportunity to build on the searching skills introduced in Pilot 1.

Appendix 2 demonstrates that the ACRL outcomes addressed by all three of the test programs were those most likely to be covered in a face-to-face tutorial for first year students in Australian university libraries. Librarians offering face-to-face tutorials in later years of undergraduate courses would almost certainly cover many of the other outcomes. The web-based tutorials used in this benchmarking exercise are suitable for students at all levels but the skills taught may need to be considerably expanded to meet the needs of the more advanced student.

As the process of evaluation against the ACRL standards is repeated across different programs, it becomes apparent that different products have different strengths and should be used appropriately. Information literacy education is sometimes planned around the need to teach a particular skill or concept. By using a method similar to that outlined in appendix 2, a librarian can quickly match the need to a package or component of a package.

While in the present study, the ACRL information literacy standards were used to assess the content of the programs being reviewed, these standards could also be used as a checklist in the design phase to ensure that information literacy programs actually cover the intended range of skills.

Quality Criteria for the Design of Web-based Instruction

From the basis of good pedagogy various authors have identified the qualities that characterise exemplary web-based instruction in higher education, and in particular, information literacy instruction. The ACRL Instruction Section Teaching Methods Committee (2000) offers Tips for Developing Effective Web-Based Library Instruction to assist librarians to develop their own web-based programs, and to enable them to better evaluate programs developed elsewhere. An earlier publication by the same committee, Evaluation Criteria for Good Web-Based Library Instruction (1998), provides an excellent list of web evaluation and design criteria, however this publication is no longer available via the web.

Dewald (1999a, 1999b) uses a range of learning theories, including adult learning theories, to devise a set of pedagogical guidelines for library instruction via the web. These guidelines make recommendations for design elements that ensure the unique capabilities of the web are used to create effective information literacy instruction. An important consideration is to assist motivation, and support the needs of learners, especially adults. Examples of ways of assisting learner motivation include the provision of immediate feedback to questions, and enabling the student to engage in real time practice in a library catalogue, database, or on the web.

The ideal program structure is one which allows the learner to follow a guided path through the course material or to choose their own path according to their needs at the time. This structure is easily accomplished via the web through such elements as an easily accessible, linked table of contents, and internal navigational aids which assist the learner to repeat sections, check a glossary, or move to other sections. An attractive example is NetTrail, from the University of California at Santa Cruz, which displays its linked table of contents as a series of locations on a topographic map.

Fast-loading pages, redirection messages to new sites, and free and open access all help to create that welcoming, first impression that encourages the learner to continue. A first page dominated by slow-loading graphics will almost certainly be a 'put-off'. General design and layout features should include clear page headings, information broken down into chunks with plenty of white space, minimal reliance on the browser navigation buttons, judicious use of images and graphics, and details about when the page was last updated. There should be an e-mail link to a librarian who can provide assistance if necessary.

Interactive exercises such as quizzes, simulations, or direct practice in a live database engage the student in deep learning where concepts rather than the mechanics of information literacy are understood. Pages that include interactive material should always provide all the instructions required to run the program.

The ACRL tips document recommends that programs use contemporary language and topics, and encourages program creators to entertain the student and not overwhelm them with excessive amounts of text. TILT, the interactive tutorial from the University of Texas Library, excels in using contemporary images, graphics and sound to keep the learner interested and focussed.

In summary, the qualities that exemplify good web-based information literacy instruction include the following dimensions:

• The program has clear learning goals and objectives
• The interface is easy to use and motivates the learner to engage with the material (graphic design, navigability, readability, site search facility)
• The learning experience is active and resources are provided in the appropriate medium (text, audio, video, simulations, quizzes, as appropriate are included)
• There is provision for online interaction with teachers and peers to facilitate a collaborative learning environment
• Both formative and summative assessment are available

From these qualities, the authors drew up a checklist of criteria that could be used to evaluate the design of web-based information literacy programs. Benchmark standards for assessing the quality of flexible learning environments developed by Griffith University's Flexible Learning Services (Griffith University, 1999) provided a useful model for the design criteria checklist. The evaluation of eGO, LRT and Pilot against the design criteria is presented in appendix 3.

The results presented in appendix 3 illustrate some interesting challenges for librarians. Criterion 4.6 shows that none of the 3 programs reviewed allows for interaction between students or between student and teacher. Koppi (1997, 248) suggests 'since communication and dialogue are at the heart of teaching and learning, technologies that facilitate the process over a network, particularly for learners distant from each other and the teacher are probably the most important". Institutions around the world are starting to trial voice activated chat software as a means of offering interactive reference services. Perhaps it is timely that these facilities be explored as a means of extending the learning opportunities offered through web-based programs.

The criterion of interactivity in the checklist needs further refinement to enable reviewers to discriminate between different levels of cognitive interaction. For example, Dewald (1999b) cites instances where web-based information literacy programs facilitate students to construct new knowledge, reflect on what they have learned and then apply that knowledge. These programs are designed to guide students through a process of synthesizing information to create their own unique and meaningful resources that can be output or saved for future reference. Such activities are more cognitively engaging than for example, pre-designed instructional activities that typically provide small chunks of information, require some form of response from the learner and then display limited pre-formulated feedback (Bannan-Ritland, Harvey and Milheim et al, 1998).

The three reviewed programs all include examples of pre-designed instructional activities. While a higher degree of interactivity is probably not necessary for introductory programs such as eGO, LRT and Pilot, librarians are beginning to explore the potential of more cognitively engaging web design to ensure successful learning of the more difficult concepts in information literacy.

While some of the qualities that are included in the design criteria checklist can be adequately rated as present or not, others such as the effectiveness of the graphic elements and the readability of the text elements could be more meaningfully measured on a calibrated scale. To improve the checklist, calibrated scales should be developed for the relevant design criteria.

Stakeholder evaluation of program quality

By measuring the programs against the established design criteria and the second ACRL standard, it was possible to gather evaluative comments from librarians and information literacy experts. It was felt that a methodology should also be developed for obtaining evaluations from primary stakeholders: students undertaking information literacy programs and academic staff involved in implementing them.

The methodology chosen was to develop a survey instrument and based on the quality criteria derived above, two versions of a questionnaire were developed. For the present study, time constraints meant that only LRT was surveyed and in this case, the 'online interaction' dimension was not surveyed because LRT does not provide this facility. The dimension of 'course relatedness' was also excluded, as LRT is by definition a generic skills program.

One version of the survey was developed for students with a more detailed version for academic staff and librarians. A similar instrument has been developed at Pennsylvania State University (2000) and this provided a useful model for the current study. Details of the questions posed in the surveys are included in appendix 4.

Student Survey Results

The student survey was administered to a group of 50 students who had undertaken LRT in second semester of 2000 and 33 responses were received. Results of the student survey are presented in chart 1 below.

Chart not available

The sample group of students was very small therefore no detailed statistical analysis can be performed on the results. Nevertheless the results show some interesting trends. The students were most satisfied with the screen layout and design, the level of language used and the personal benefits they gained from undertaking the program. Other elements that rated well were the simulations and the organisation of the material. Students were least satisfied with the assessment in that they did not find those activities easy to complete. Students were also uncomfortable with the concept of recommending the program to others. The capacity to leave the program and return to the same point later was a third element rated poorly by the student group.

It is obvious from chart 1 that web-based programs suit the learning styles of some individuals. While a percentage of users prefer the on campus environment of watching a demonstration, listening to descriptive detail and asking questions, there is another group who are happy to work independently and individually. Web-based programs allow the student to choose the time, place and manner in which he/she uses the program. Progress is at the pace of the individual and particular modules can be ignored or repeated as often as desired. Students can also rely on web-based programs to revise concepts and skills as they are needed. It has also been argued that information skills taught at the point of most need are remembered, and more likely to change lifelong information seeking behaviour (Richardson, 1994).

Staff survey results

The staff survey was completed by a small group of 7 staff from 3 institutions (Griffith, CQU and USQ). The results of the staff survey are presented in chart 2. Once again the small size of the survey group means that no statistical analysis can be performed. The trends that emerged in the staff survey were largely consistent with the student survey. It should be noted that staff were asked some additional questions about the clarity of the learning goals and the structure of the program.

Like students, staff were most satisfied with the organisation and readability of the material. They were also satisfied with the provision of a clear outline for the structure of the program, a question not asked of students. They were moderately satisfied with most other aspects of the program except for the facility for students to leave the tutorial and return to the same point later. Students expressed the same opinion about this dimension.

Chart not available

For the elements of the program that require the Shockwave™ plugin (the simulations and the assessment activities), several of the staff reviewers were unable to obtain access. Thus it is not possible to draw any conclusions about those aspects of the tutorial based on the extremely limited staff responses to those questions.

It should be noted that the student group were accessing LRT on campus at Griffith University and therefore the question on time to load pages was not included. Several of the staff reviewers were accessing the program from remote sites and time to load pages was an issue raised by those reviewers. This is a concern that will need to be addressed by the program's developers if off-campus access is to be supported.

Several staff made useful observations about LRT in the comments box. One reviewer was very positive about the simplicity of the program but questioned the need for this level of simplicity in a tertiary education environment. Another person found the activities, simulations and quizzes to be the best feature of the program but was disappointed that some of the quiz questions had very leading prompts.

The success of web-based information literacy instruction is vitally dependent on students' experience of the learning environment created by programs such as Pilot, eGO and LRT. Furthermore, the opinions of academic staff and librarians recommending such programs to their students must be considered. Some of the elements being evaluated are subject to individual preferences and feedback from a wider audience than the expert reviewer must be sought. For example, any assessment of dimensions such as the quality of the graphics and the readability of text elements can be a very subjective judgement.

The feedback obtained from the survey of students and staff validates and extends the assessment that was carried out using the checklist of design criteria. As more survey data is gathered over time, the reliability of the cumulated student and staff evaluation will improve thus providing developers, academic staff and students with assurance of the product quality.

Other dilemmas in designing web-based information literacy programs

In addition to the aspects of web-based instruction discussed above, there are some idiosyncrasies of using the web as a medium for information literacy education. Issues such as whether programs should be generic or discipline specific, the level of institutional specificity to be adopted, and how to allocate ongoing resources to the staff intensive process of web program development, all need careful consideration.

Course-related information literacy education

It is widely agreed that the most effective information literacy instruction is that which is directly relevant to particular courses and can be provided at the time the students require it. While web-based programs are ideally suited to meeting the second criterion in that most are readily accessible, the first criterion of course-relatedness raises interesting issues.

Although generic programs are usually cheaper to produce, can be shared by more institutions and have wide versatility, they are not always recognised by users as useful, interesting or relevant. On the other hand discipline specific or course related programs are truly relevant to a select group of users. The choice largely depends on the teaching objectives of the institution and whether the approach to information literacy instruction is generic or discipline based. As teaching objectives change the impact on web-based products can be considerable, as it implies changes to both content and instructional design.

The three programs under review in this paper are all designed as generic, stand alone programs. In practice however, a generic program may be implemented within the context of a student's normal coursework and through reinforcement by teaching staff, may become course-related. For example, a generic web-based information literacy package may be presented within the tutorial program of a subject and may even be facilitated in class time by teaching staff and librarians. In this context the package becomes course or assignment specific.

Institutional specificity

Another consideration is the extent to which programs should teach the specific products available at the home institution. For example, staff at Griffith made a decision to demonstrate the Griffith library catalogue and use live links to the catalogue to reinforce the learning. Although this is instructionally desirable and seen as a good design principle, it means the product has little relevance outside Griffith and the high development costs can not be reduced by sharing the product with other institutions.

Resource issues

In choosing to develop, use and maintain web-based programs careful consideration needs to be given to resource implications. All librarians have realised the impact that face-to-face instruction has on the commitment of staff time and it is sometimes easy to assume a web-based product will reduce this commitment. There is little evidence to suggest that staff time associated with development of content, especially when the content is constantly evolving, is any less than that needed to support a face-to-face program. Likewise, design issues are important and as the content of a package changes, staff time to develop an appropriate, instructionally sound medium is considerable.

Conclusion

While the web has the potential to be a powerful medium for instruction, there is still significant uncertainty about how the various media elements should be used and what sorts of content are suited to the medium. The evaluation techniques described in this paper (the process of matching content with the ACRL standards, the design criteria checklist and the survey instruments) together provide a framework for assessing the design and content of contemporary Australian web-based information literacy programs. These evaluation methodologies are themselves based on sound educational practice and current adult learning theories. Such techniques go some way towards assuring academics, librarians and students that the web is being used to its best advantage in providing quality learning environments for information literacy education.

Appendix 1: Description of eGO, LRT and Pilot

eGO

eGO is the University of Southern Queensland Library's Internet literacy training program, an interactive tutorial which offers exercises in live sites and detailed instructions and assistance on:

• searching strategies;
• how to use the catalogue;
• how to search databases;
• finding relevant information on the Web;
• referencing;
• evaluating your sources, and
• communicating on the internet via e-mail, newsgroups and chat.

eGO contains practice sessions or quizzes to test skills learned, and assistance in completing the activities.

The Library Research Tutorial: An Information Literacy Skills Program

The Library Research Tutorial (LRT) provides students with instruction, on-line practice, feedback and assessment in generic library research and information retrieval skills. The Tutorial is designed to help students gain skills in finding information on their topic of research within Griffith University Library and beyond.

The program comprises seven modules:

• Thinking about the topic
• What sort of information is needed
• Introduction to databases
• Using the library catalogue (telnet and web)
• Other electronic databases
• World Wide Web resources
• Evaluating resources

Pilot: Your Information Navigator

Pilot is Queensland University of Technology's online information literacy tutorial encompassing all facets of information seeking and evaluation. The program comprises 9 modules:

• Defining you topic
• Determining your information needs
• Locating and retrieving relevant information
• Researching with electronic tools
• Evaluating information
• Organising your information
• Communicating with technology
• Using information appropriately
• Overview of QUT library

Appendix 2: Evaluation of eGO, LRT and Pilot against ACRL Information Literacy Competency Standard Number 2

Appendix 3: Design criteria for effective web-based information literacy instruction

Appendix 4: Details of staff and student surveys

For both the student and staff surveys, participants were asked to respond by indicating their level of agreement to the statements below. They were offered a 5 point Likert scale (Strongly disagree, Disagree, Not sure, Agree, Strongly Agree). They were also given space to make any other comments they wished. Students were asked some additional demographic questions (age, gender, level of education and language background).

Staff survey

1. There is a clear outline of the Tutorial content and structure

2. The learning goals and outcomes of LRT are well explained.

3. The screen layout and design makes it easy to work through the Tutorial.

4. The graphics enhance the learning experience.

5. The time to start up LRT and load the homepage is acceptable.

6. Navigation through the Tutorial is easy.

7. The design makes it easy for the student to leave the Tutorial and return to the same point later.

8. The content is well organised and follows a logical sequence.

9. It is easy to skip areas in which instruction is not needed.

10. The content is easy to understand.

11. There is too much information on the screen.

12. The simulations allow students a good opportunity to practice skills.

13. The Tutorial should be more interactive.

14. The assessment activities are a good test of the student's learning.

15. The feedback given in the simulations and assessment activities is helpful for the student.

16. I would recommend LRT to students.

Student Survey

1. Navigation through the Tutorial was easy.

2. The screen layout and design makes it easy to work through LRT.

3. The graphics enhanced the information presented.

4. The design made it easy for me to leave the Tutorial and return to the same point later.

5. The content was well organised and followed a logical sequence.

6. It was easy to skip areas in which I did not need instruction.

7. The information covered was easy to understand.

8. The level of language used was easy for me to understand.

9. I found the catalogue and database simulations helpful.

10. The assessment activities were easy to complete.

11. I found the feedback given in the assessment activities helpful.

12. I would recommend LRT to other students.

13. LRT helped me to use the Library more effectively.

Bibliography

Association of College and Research Libraries 2000, Information Literacy Competency Standards for Higher Education, [Online], Available: http://www.ala.org/acrl/ilcomstan.html [Accessed 6 October 2000].

Association of College and Research Libraries Teaching Methods Committee 2000, Tips for Developing Effective Web-Based Library Instruction, [Online], Available: http://www.lib.vt.edu/istm/WebTutorialsTips.html [Accessed 18 October 2000].

Association of College and Research Libraries Teaching Methods Committee 2000b, Evaluation Criteria for Good Web-Based Library Instruction, [Online], Available: http://www.bk.psu.edu/academic/library/istm/criteria.html [Accessed 20 June 2000 but since withdrawn].

Bannan-Ritland, Brenda; Harvey, Douglas M. and Milheim, William D. 1998, A general framework for the development of web-based instruction. Educational Media International, 35(2), 77-81.

Bruce, Christine and Candy, Philip C 1994, 'Higher education contributions to information literacy education: towards a checklist for evaluating curriculum and institutional culture', Higher Education in Transition: a Collection of Papers Presented at the Twentieth Annual National Conference, Higher Education and Research Development Society of Australasia,Cook ACT, [CD-ROM]

Dewald, Nancy H. 1999a, Transporting good library instruction practices into the web environment: An analysis of online tutorials. Journal of Academic Librarianship, 25(1), 26-32.

Dewald, Nancy H. 1999b, Web-based library instruction: What is good pedagogy? Information Technology and Libraries, 18(1), 26-31.

Griffith University 1999, Appendix 3 "Benchmarking Flexible Learning Environments, Products, and Services" - Commercialisation Report - Phase 2 (Griffith Flexible Learning Services). Brisbane.

Griffith University 2000, Library Research Tutorial [Online] Available: http://www4.gu.edu.au/shr/lrt/ [Accessed 2 October 2000]

Koppi, A.J., Lublin, J.R. and Chaloupka, M.J. 1997, Effective teaching and learning in a high-tech environment. Innovations in Education and Training International, 34 (4), 245-251.

Pennsylvania State University 2000, Training Wheels Survey [Online] Available: http://www.de2.psu.edu/tutorials/trainingwheels/surveyintro.html [Accessed 20 June 2000]

Queensland University of Technology 1999, Pilot [Online] Available: http://wwwlib.qut.edu.au/elearn/tutorial.html [Accessed 20 October 2000]

Richardson, Gregg 1994, Computer-assisted library instruction? Consider your resources, commitment and needs. Research Strategies, 12(1),45-55.

University of California, Santa Cruz 1998, NetTrail [Online] Available: http://www2.ucsc.edu/nettrail/master/ [Accessed 19 October 2000]

University of Southern Queensland 2000, eGO [Online] Available: http://www.usq.edu.au/b-eGO [Accessed 2 October 2000]

The University of Texas System Digital Library 2000, Tilt [Online] Available: http://tilt.lib.utsystem.edu/ [Accessed 19 October 2000]