ALIA home | Online home | search | papers | keynotes | program | sponsors | satellite events
|
Fiona Salisbury and Jenny Ellis
Arts Learning Resources Team
Teaching, Learning and Research Support Department
Information Division
University of Melbourne
During 2002 the University of Melbourne released several landmark documents that have important implications for information literacy programs. In particular, the Nine Principles Guiding Teaching and Learning in the University of Melbourne was published in April 2002 with the imprimatur of Academic Board as 'a framework for a first-class teaching and learning environment.' [1] Principle eight deals with the objectives of providing 'premium quality learning resources and technologies' and it includes the following statement:
'It is essential for Melbourne graduates to develop the skills of independent information searching, evaluation and utilisation using all available sources of information.'[2]
The yet to be published Draft University Teaching and Learning Management Plan September 2002 also urges the 'integration of information literacy into the curriculum.'[3] Furthermore, in 2002, Academic Board endorsed the broadly-based CAUL Information Literacy Standards[4] as a 'useful guide to curriculum design.'[5] At Departmental level, the generic skills statements of many subjects refer to the achievement of information literacy goals.[6] Hence information literacy is firmly on the University agenda and there is a clear organisational expectation that University of Melbourne students should be sophisticated users of information by the time that they complete their degrees.
For undergraduates, these objectives cannot be met without the implementation of broadly based programs that are deliberately designed to achieve information literacy outcomes. Programs need to be widely available to all students and based upon their current research needs, ideally allowing students to sequentially build upon their information skills as they progress through University. To ensure that the information literacy goals of the University and its student community are met, these programs need careful scrutiny and evaluation.
At the University of Melbourne, the development and delivery of these information literacy programs is primarily the province of the Learning Resources Teams. There are three discipline-based teams, each composed of four information specialists, one for Arts; Sciences; and Social Sciences. A web publishing team supports the teams. Learning Resources Services is not located within the Library structure although both are located in the Information Division. Instead the section is located in the Teaching, Learning and Research Support Department of the Information Division, reflecting its priority to embed information literacy training in the curriculum and to embrace new technologies for teaching and learning. This paper reflects the experience of the Arts Learning Resources Team that is charged with embedding information literacy into the curriculum in the Faculty of Arts.
Since its inception in the 2001 academic year the Arts Team has made great strides in increasing the information literacy opportunities offered to Art students in the Faculty. In 2002 we reached 5300 students in a total of 172 interventions across 66 subjects. These interventions were all based on sessions that were designed and delivered in collaboration with academic staff as part of the curriculum for specific subjects. They may have taken the form of lectures to large groups, tutorials, demonstrations, hand-on computer sessions and individual consultations to Honours students. Subject-specific online tutorials were also developed in some areas. Most of the interventions were formally timetabled within the academic timetable although some required voluntary attendance.
Even though the team can demonstrate substantial involvement within the Faculty, it is still difficult to confidently measure the impact on the information literacy levels of Arts students. We are constantly faced with the complex task of ascertaining who has learnt what. Unlike most faculties, Arts students do not enrol in any core subjects so it is quite possible that some students are receiving two or three information literacy sessions in any one year while others receive none at all. And since we teach into the curriculum, learning outcomes are determined to a large extent by curriculum requirements so that information literacy outcomes for the undergraduate years are not consistent across the Faculty. This makes it difficult to evaluate the effectiveness of information literacy programs across the Faculty.
In response to these issues the Arts Learning Resources Team developed a framework for mapping the integration of information literacy skills across the Faculty of Arts resulting in a document entitled Integration of Information Literacy Skills: A Framework for the Faculty of Arts 2002. The framework, which is aligned with the Council of Australian University Librarians (CAUL) Information Literacy Standards, is now in a trial phase. It identifies common outcomes that are deemed necessary for information literate Arts students. The outcomes are grouped into three levels that roughly correspond to the undergraduate years. Hence, learning outcomes at level one of the Framework are most suitable for first or second year subjects or subjects that introduce library research. By the end of their degree, it is expected that Arts students will have achieved the outcomes specified in the Framework and will have met many of the in formation literacy objectives expected of the University graduate.
Figure 1 - Translating University information literacy objectives via Arts IL Framework
The development of the Arts Framework was critical for the formation of an evaluation strategy for the Faculty of Arts. It enabled learning outcomes of individual sessions across the Faculty to be matched to common outcomes in the framework. The achievement of these common outcomes can them be systematically evaluated. In 2002 we began this systematic evaluation of our information literacy program, planning a process that will evolve in several defined stages. This process will hopefully enable the Arts team to make informed judgments about the effectiveness of programs in reaching university objectives and information literacy needs of Arts students.
Our evaluation practice was informed by the evaluation model devised by Donald Kirkpatrick.[7] This model is commonly used in the literature relating to training and it is applicable to measurement of the effectiveness of skills training. It is a particularly accessible model and can be readily adapted to evaluate some of the more concrete skills of information retrieval that are routinely taught at first year level.
The Kirkpatrick model outlines four levels of evaluation - Reaction, Learning, Behaviour and Results.[8] Level one evaluation deals with the reactions of participants, probing their response to the training and the trainer and their perceptions of its success in meeting their expectations.
Level two evaluation refers to learning or 'the extent to which participants change attitudes, improve knowledge, and/or increase skills as a result of attending the program.'[9] Level three relates to measuring changes in long-term behaviour in individuals as a result of attending the program. For example, a level three evaluation would follow up whether or not the student had actually transferred the skills to apply them to later research tasks and assignments. Level four evaluation relates to the impact or benefit of the training at an organisational level.
The following table (substantially reproduced from Forsyth et.al., 1995)[10], outlines the four levels of the Kirkpatrick model:
| Level | Evaluation context | Data collection |
| One - Reaction | A measurement of the learner's feelings and opinions about the course just completed | Eg. Methods of instruction, course content, learning materials, facilities. |
| Two - Learning | A measurement of what has been learned as a result of the learning event. Examines facts, skills and attitudes that the learner has gained from the event. | Collects information related to the learner's achievement of the learning event's objectives. |
| Three - Behaviour | A measurement of the behavioural changes occurring as a result of the learning event just completed. | Collects information regarding the individual learner's performance following the learning event. It looks for the impact of changes in a new situation. |
| Four - Results | A measurement of the overall impact of an innovation on the institutional environment | Relates to costs, morale, productivity on a total institutional basis. |
Level one evaluation is useful for evaluation of library instruction because it gives a window into the students' satisfaction with the session. Library literature reveals that libraries routinely conduct this level of evaluation based on questionnaires or 'happy sheets' distributed at the end of the session.[11] It is the most commonly implemented form of evaluation for library information literacy session.[12] The Arts Team routinely distributes happy sheets and will continue to do so because it is useful to know whether or not students found the training successful; whether they approved of the venue and training materials, and whether or not they found the presentation to be logically presented and easy to follow. This is all useful feedback and it allows improvements to be made to the sessions. But it is worthwhile to note that a positive response on a 'happy sheet' does not necessarily indicate that effective learning has taken place.[13]
For this reason the Arts team decided that it was necessary to move beyond an evaluation process that only considered participant perceptions of a session. We recognized a need to at least attempt to verify that students actually did acquire skills and knowledge as a result of our efforts. This required evaluation to level two of the Kirkpatrick model.
Measuring what has been learned as a result of a learning event is not easy for information professionals to conduct. The general inadequacies of evaluation in the library and information sector is explained in the literature by insufficient budget, lack of expertise, and lack of methods and tools.[14] All of these are contributing factors, but even more significant is the insufficient time available in what Ragains calls 'one-shot' intervention in a subject.[15]
Information skills classes occupy only a small part of the timetable and often there is only one intervention per semester in a subject. This makes it difficult to follow-up the effectiveness of learning experiences. If information skills were all formally assessed this would not be such a problem but assessment tasks are still not the norm. Hence we needed to develop a method of evaluating learning that could be conducted within these constraints. All of these factors were taken into account in developing the objectives of the evaluation.
The objectives of this part of the evaluation process were basically formative. They were:
A pre-test and post-test methodology was used for this research to determine whether learning objectives were achieved as a result of the learning experience. This took the form of a paper-and pencil test formulated according to learning outcomes that were common to the Arts Framework. The pre-test was required to ascertain prior knowledge of students. The post-test was administered immediately following the sessions to ensure that any changes in knowledge could be attributed to the intervening session. The use of this methodology is considered to be an appropriate method for testing increased knowledge and skills as a result of a learning event.[16]
The questions on the tests were closely related to the content of the session. Most of the questions were in multiple choice or Yes/No format so that the results were easily quantified. However some qualitative comment was also invited. The tests were very short so that it could be squeezed into the start and finish of an hour-long session that already had a crowded agenda.
Advantages of the pre-test/post-test methodology:
Limitations of pre-test/post-test methodology:
This pilot evaluation was carried out in three first year Arts subjects during Semester one, 2002 and involved a total of 282 students. Students were enrolled in Australian History, Medieval History, and Film Theory. Students were not obliged to complete the tests, but response was close to 100 per cent.
The information literacy sessions were offered to tutorial groups of a similar size each including approximately 15 students. Different modes of delivery were used:
Although the sessions were subject specific they all included some common learning outcomes that could be aligned to the Level one of the Arts Information Literacy Framework. This level outlines learning outcomes that are considered necessary for students attempting first or second year Arts subjects or subjects that introduce library research. The pre and post-test measured student knowledge in 1.1 and 1.3 of this framework.
| Arts Information Literacy Framework Level 1 Learning outcomes |
| 1.1 Interpret a reading list |
| 1.2 Analyse a topic |
| 1.3 Construct a search strategy |
| 1.4 Search the library catalogue effectively |
| 1.5 Search a relevant index or database effectively using basic search strategies |
| 1.6 Physically locate items in the university library system |
| 1.7 Critically evaluate search results |
| 1.8 Consistently use appropriate citation styles for a range of resources |
Questions designed to measure outcome 1.1 of the Arts framework deal with interpreting a reading list - recognizing citations to books, journal articles, chapters in books, etc. They further determined whether students had the knowledge able to locate the journal in the library catalogue. For example, students in Film theory were given the following citation and problem:
Smith, G. (1997) 'Mind over Matter,' Film Comment, vol.33, no.2, pp.14-29
Questions designed to measure outcome 1.3 probed the students' familiarity with Boolean search strategies before and after the session by asking the students to complete a search statement with the appropriate Boolean operator. For example students studying Medieval History might be presented with the following problem:
If you searched the Library catalogue for information about the sack of ROME by the GOTHS, which Boolean operator would you use to link the key concepts?
The same question was repeated in the post-test and any improvement in understanding was noted. These search statements were adapted to reflect different subject content, but the general outline remained similar.
The formulation of questions was a complex task resulting in several iterations. The questions were initially piloted with selected undergraduate students and library colleagues. Even so, it was necessary to re-phrase the Boolean logic questions after experience with the first few groups because the task was demonstrably too difficult. This process of refining the questions is ongoing.
Students were also asked a question about attitudes, specifically being asked to rate their confidence in using Boolean operators following the training. They were also offered the opportunity to add qualitative comments about the session.
In general, the results are strikingly similar for classes that were delivered online or using face-to-face methods. This finding has also been reported in other comparative studies from library and education disciplines.[17]
In the pre-test the majority of all students (73 per cent) could identify the journal citation correctly. However, only 23 per cent of the total group understood how to locate the journal in the library catalogue. This data is useful in identifying gaps in student entry-level knowledge. It demonstrates that the majority of students have very limited knowledge when it comes to finding items given on a reading list. These results also reflect anecdotal evidence from information desk staff that first year students have no idea how to go about locating citations on their reading lists in the library.
Figure 2 - Journal article citations - First year students' entry-level knowledge - pre-test
The three questions relating to interpreting a reading list were repeated in the post-test. In each group there was an improvement in the number of students who were able to recognise the journal citation. In the Film Theory there was a 16 per cent improvement. Similarly there was a 14 per cent improvement in the Medieval History group and in the case of Australian history there was a 48 per cent improvement. In the Australian history group 59 per cent of students were able to recognise the journal article in the pre-test but 88 per cent were able to recognise the journal article in the post-test. After their session 91 per cent of the total group were able to identify the journal citation.
Figure 3 - Ability to recognise journal citations - pre-test and post-test
There was also and increase in understanding of the strategy needed to search for the citation in the catalogue was demonstrated in the post-test. Although the improvement was significant, the overall result was disappointing since 57 per cent of students still did not achieve the outcome. In each group the improvement in the number of students who gained this knowledge is significant but overall there is not a high level of understanding of the process involved.
These results indicate that for new students understanding how to locate a journal article citation in the library is a complex and new concept that is difficult to grasp in a one-hour tutorial. Regardless of the method of instruction and format of the tutorial, the majority of students were not able to understand this complex concept after one single intervention. These results do not reflect any significant difference that could be attributed to modes of delivery but rather highlight a difficulty in understanding base level knowledge. This needs to be addressed through development of a sequenced program if students are to have a firm foundation for first year and beyond.
Figure 4 - Recognise and search for journal citation - pre and post-test
Students were asked considered to have met learning outcome 1.3 in the Arts IL Framework if they were able to correctly place both AND and OR in the given search statement.
In the pre-test 26 per cent of the total group were able to complete this task correctly. In the post-test there was an improvement in the number of students who were able to correctly use Boolean operators. In Medieval History and Australian History the improvement was significant but still over 50 per cent of the groups did not achieve the outcome. Again, this reflects the difficulty students have in grasping a difficult concept like using Boolean operators to construct a search statement in a one-hour session.
Figure 5 - Boolean operators - pre-test and post-test
When the students were given a memory trigger or cue in the Film Theory post-test, the results were substantially improved. In this question, students were brackets were already inserted at the point where they had to select the OR operator. It seems that the brackets may have triggered most students to recall what had been demonstrated in the tutorial.
Figure 6 - Boolean operators - Film Theory students - pre-test and post-test
It appears that students commence their Arts degree at this university with very limited knowledge of library research processes. This has been a particularly illuminating result of the evaluation process since it gave us a window into the knowledge that students have when they start an Arts degree. This will be built into future needs analysis. It has been useful to be able to present this concrete data to academic staff.
There is no doubt that information literacy interventions in first year subjects contributed to a substantial improvement in information skills. In some areas the training was very successful, achieving a high standard of knowledge across three groups. However high levels of achievement tended to be prominent in lower order skills such as recognising a journal article citation. When conceptual understanding became more complex, students exhibited a much lower rate of improvement as a result of the session. This indicates that difficult concepts such as Boolean searching need further reinforcement. It points to the inadequacy of a single information literacy session in a semester and suggests that opportunities need to be made for further interventions. Failure to grasp a complex concept in one session is hardly surprising result. Fortunately as Kunkel philosophically points out, first year students still have time ahead of them at university to develop these skills. [18]
We can also reflect on our teaching practice in order to improve the learning experience for students. Our lesson plans are crowded and it appears that we try to achieve more learning outcomes in one session than is realistic. As a result of the evaluation we incorporated this finding into the development of a new online tutorial. Art History Research Detective is a tightly focused tutorial, which concentrates solely on interpreting a reading list. Preliminary findings indicate that the effectiveness of this tutorial was comparatively high since 97 per cent of students were able to locate the journal article in the post-test. This suggests that fewer learning outcomes per intervention would probably be a successful strategy but this would mean that more sessions would need to be scheduled.
One way of increasing the number of information literacy opportunities offered to students would be to use a combination of modes of delivery. We may only be given one opportunity for face-to-face delivery during a semester but there is room for online components. This would cater for diverse learning styles and increase the number of interventions possible in a semester. This might produce more satisfactory outcomes for students as our study revealed that no one method of delivery was ideal when there was only one intervention.
The evaluation process has been built into the information literacy program of the Arts team from the outset and it is being sequentially expanded each year. Initially the evaluation consisted of the traditional questionnaire or 'happy sheet' that is handed out at the end of a training session. These were useful in providing information about the general satisfaction of participants and we continue to use this evaluation method on a selective basis. However this was not sufficient to ensure that we had achieved the information literacy objectives of the university and the students.
For this reason we embarked upon an evaluation of the learning that takes place in individual sessions. According to Kirkpatrick, this is the second step in the evaluation process since learning is a prerequisite for change in behaviour. Learning must occur before students can transfer knowledge and skills to different situations. The process has been very beneficial for the Arts team since it yielded valuable insights into the knowledge that our students have when they start at University. It has, without doubt, also been an iterative process for the evaluation results have suggested improvements and future directions for our teaching methods and mode of delivery.
This evaluation method using pre-test and post-test has also proved to be realistic for information literacy staff in that it is both cost-effective and able to be completed within the time constraints of the traditional one hour library skills session.
Armed with the results of the 2002 study we will continue to improve this methodology and re-apply it next year across a broader cross-section of subjects. However we will also move to the next level of evaluation and try to follow-up the impact of information literacy training on the long-term information habits of students. A questionnaire is planned for this purpose. It will be a self-reporting tool asking students to report on their actual behaviours and will be accompanied by focus groups. Through out efforts in evaluation, and with time and persistence, we trust that we will eventually be able to make informed judgements about the achievement of information literacy outcomes across the Faculty.
Caravello, Patti Schifter, Judith Herschman, and Eleanor Mitchell. 'Assessing Information Literacy of Undergraduates: Reports from the UCLA Library's Information Competencies Survey Project,' in ACRL Tenth National Conference, March 15-18, 2001, 193-202. Denver, Colorado: 2001.
Cotton, Julie. The Theory of Assessment : An Introduction. London ; Philadelphia: Kogan Page, 1995.
Dow, Kwong Lee. 'Teaching and Learning Management Plan.' 35. Melbourne: The University of Melbourne, 2002.
Eseryel, Deniz. 'Approaches to Evaluation of Training: Theory and Practice.' Educational Technology and Society 5, no. 2 (2002): 93-98.
Forsyth, Ian, David Stevens, and Alan Jolliffe. Evaluating a Course : Practical Strategies for Teachers, Lecturers and Trainers, The Complete Guide to Teaching a Course. London: Kogan Page, 1995.
Hong, Kian-Sam. 'Web-Based Learning Environments: Observations from a Web-Based Course in a Malaysian Context.' AJCT 17, no. 3 (2001): 14-17.
Information Literacy Standards. 1st ed. Canberra: Council of Australian University Librarians, 2001.
James, Richard, and Gabrielle Baldwin. Nine Principles Guiding Teaching and Learning in the Univeristy of Melbourne. Melbourne: University of Melbourne, 2002.
Kirkpatrick, Donald L. Evaluating Training Programs : The Four Levels. 2nd ed. San Francisco, Calif.: Berrett-Koehler Publishers, 1998.
Kunkel, Lilith, Susan Weaver, and Kim Cook. 'What Do They Know?: An Assessment of Undergraduate Library Skills.' Journal of Academic Librarianship 22, no. 6 (1996): 430-34.
McPhee, Peter. 'News from Acadmemic Board.' Uni News, 6 May (2002): http://www.unimelb.edu.au/ExtRels/Media/UN/archive/2002/710/newsfromAB.html.
O'Connor, Lisa, Carolyn Radcliff, and Julie Gedeon. 'Assessing Information Literacy Skills: Developing a Standardized Instrument for Institutional and Longitudinal Measurement.' in ACRL Tenth National Conference, March 15-18, 2001, 163-74. Denver, Colorado, 2001.
Phillips, Jack J. Implementing Evaluation Systems and Processes, In Action. Alexandria, VA: American Society for Training and Development, 1998.
Ragains, Patrick. 'Evaluation of Academic Librarians' Instructional Performance: Report of a National Survey.' Research Strategies 15, no. 3 (1997): 159-75.
Shonrock, Diana D. Evaluating Library Instruction : Sample Questions, Forms, and Strategies for Practical Use. Chicago: American Library Association, 1996.
The Australian Centre, University of Melbourne. 'Generic Skills for Arts Students.' http://wwwaustralian.unimelb.edu.au/courses/undergrad/genericskills.html, 2002.
Wilson, Carol. 'A Tale of Two Classes.' Educause Quarterly, no. 2 (2002): 72.
[1] Richard James and Gabrielle Baldwin, Nine Principles Guiding Teaching and Learning in the Univeristy of Melbourne (Melbourne: University of Melbourne, Final draft 3/09/2002).
[2] Ibid. 18
[3] Kwong Lee Dow, 'Teaching and Learning Management Plan,' (Melbourne: The University of Melbourne, 2002).
[4] Peter McPhee, 'News from Acadmemic Board,' Uni News, no. 6 May (2002).
[5] Information Literacy Standards, 1st ed. (Canberra: Council of Australian University Librarians, 2001).
[6] For example see the Generic skills statement for The Australian Centre at http://wwwaustralian.unimelb.edu.au/courses/undergrad/genericskills.html
[7] Donald L. Kirkpatrick, Evaluating Training Programs : The Four Levels, 2nd ed. (San Francisco, Calif.: Berrett-Koehler Publishers, 1998).
[8] Ibid. 19
[9] Ibid. 20, 39
[10] Ian Forsyth, David Stevens, and Alan Jolliffe, Evaluating a Course : Practical Strategies for Teachers, Lecturers and Trainers, The Complete Guide to Teaching a Course. (London: Kogan Page, 1995). 10
[11] Kirkpatrick, Evaluating Training Programs : The Four Levels.19-20; Jack J. Phillips, Implementing Evaluation Systems and Processes, In Action. (Alexandria, VA: American Society for Training and Development, 1998). 2, Deniz Eseryel, 'Approaches to Evaluation of Training: Theory and Practice,' Educational Technology and Society 5, no. 2 (2002). 94, Forsyth, Stevens, and Jolliffe, Evaluating a Course : Practical Strategies for Teachers, Lecturers and Trainers. 10-19
[12] Patrick Ragains, 'Evaluation of Academic Librarians' Instructional Performance: Report of a National Survey,' Research Strategies 15, no. 3 (1997). 64, Diana D. Shonrock, Evaluating Library Instruction : Sample Questions, Forms, and Strategies for Practical Use (Chicago: American Library Association, 1996). The latter book is almost entirely devoted to devising samples and strategies for completing level 1 evaluations.
[13] Kirkpatrick, Evaluating Training Programs : The Four Levels. 25ff; Patti Schifter Caravello, Judith Herschman, and Eleanor Mitchell. 'Assessing Information Literacy of Undergraduates: Reports from the UCLA Library's Information Competencies Survey Project,' in ACRL Tenth National Conference, March 15-18, 2001 (Denver, Colorado: 2001). 193
[14] Eseryel, 'Approaches to Evaluation of Training: Theory and Practice.' 93; Lisa O'Connor, Carolyn Radcliff, and Julie Gedeon, 'Assessing Information Literacy Skills: Developing a Standardized Instrument for Institutional and Longitudinal Measurement,' in ACRL Tenth National Conference, March 15-18, 2001 (Denver, Colorado: 2001). 164
[15] Ragains, 'Evaluation of Academic Librarians' Instructional Performance: Report of a National Survey.' 160
[16] Forsyth, Stevens, and Jolliffe, Evaluating a Course : Practical Strategies for Teachers, Lecturers and Trainers.16, Kirkpatrick, Evaluating Training Programs : The Four Levels. 47, Julie Cotton, The Theory of Assessment : An Introduction. London ; Philadelphia: Kogan Page, 1995. 15.
[17] Carol Wilson, 'A Tale of Two Classes,' Educause Quarterly, no. 2 (2002). 72 ; Kian-Sam Hong, 'Web-Based Learning Environments: Observations from a Web-Based Course in a Malaysian Context,' AJCT 17, no. 3 (2001).
[18] Kunkel, Weaver, and Cook, 'What Do They Know?: An Assessment of Undergraduate Library Skills.'