Assessment

Philosophy and process

unanimously updated and reaffirmed by LIS faculty 28 August 2020

Overview:​​ Not all of our strengths can be captured in numbers.  For program-level assessment, we prefer an approach where many voices can be heard and considered, over a longer period of time than a single semester.  When issues arise with a course, instructor or student, those are addressed individually by the LIS chair.  When issues persist and recur across multiple courses and semesters, then we feel we are justified in claiming an area of strength, or in devoting resources to addressing areas where improvement is needed.

Lived experience:​​  The most organic form of assessment is our lived experience, where we enact our values of aloha, ʻohana and kuleana.  While it would be inappropriate to attempt to quantify these experiences, our graduating student surveys tell us that the sense of community within the LIS Program is by far what our students value most.  As part of our teaching, research, service and informal interactions, we share our experiences throughout the year, documented in meeting minutes and year-end reports.

Critical refusal: While acknowledging the usefulness of assessment metrics for some purposes, our philosophy also encompasses critical refusal of harmful data regimes, as summarized in the Feminist Manifest-No.

Course evaluations: ​​As a formative assessment of student learning and faculty instruction, each student evaluates each course in terms of content, workload, delivery and assessment methods, and instructor effectiveness.  Numerical and anecdotal data are collected by the Program Coordinator and reviewed by the chair each semester.  Instructors whose evaluations are lower than 4/5 meet with the chair to debrief.  Adjuncts who consistently fail to reach the 4/5 standard, or who receive very low ratings, are not invited back to teach.

Course SLO assessments:​​ As a formative assessment of program SLOs, each instructor identifies the number of students who exceeded, met, approached or did not meet the standards associated with the primary SLO of the course via a major assignment, or a set of related assignments.  This is a more direct, formative measure of SLO mastery than an overall course grade.  The chair collects and reviews these assessments at the end of each semester (adjunct faculty) or academic year (full-time faculty), and summarizes compiled student achievement by SLO back to the faculty in the following year’s planning session.

Culminating experience assessments:​​ As a summative assessment of program SLOs, students taking the oral comprehensive exam respond to four scenarios associated with the 6 program SLOs.  The average of two faculty members’ scores must be above 5/7 on every scenario in order to pass. Similarly, students completing the ePortfolio produce an artifact and reflective essay associated with each SLO, which is scored by two faculty members.  In both cases, results are aggregated by the chair and reported back to the faculty as a summative measure of student achievement by SLO.

Graduating student surveys:​​ A summative assessment of the overall student experience, given each semester to graduating students

Alumni surveys:​​ A reflective assessment of student experience and professional skills needed and acquired administered every 5 years.

Employer surveys:​​ A merged assessment of UHM LIS alumni skills, job performance and needed skills within each employer organization, administered every 5 years.

Committee work and event documentation: ​​Each faculty member leads one or more standing committees, task forces or initiatives.  As a formative assessment of student experience, stakeholder engagement and faculty service, each faculty member summarizes their groups’ process, student and stakeholder engagement and accomplishments in a year-end report.

Year-end meeting.​​  To contextualize assessment data and to systematically collect and compile stories, reflections and any other issues, the LIS Associate Chair collects year-end reports and meets with each faculty member, either in person or virtually, at the end of the academic year, to review and elicit their reflections on:

  • Their course SLO assessments, to contextualize the numerical data and discuss any opportunities or challenges with their courses.
  • Their year-end report of committee work, stakeholder engagement, metrics etc.
  • Their assessment of the performance of the LIS Chair, Program Coordinator and staff.

The Associate Chair summarizes faculty discussions and reflections (any faculty comments, critical or otherwise, may be anonymized at the discretion of the faculty member providing them), and discusses the results in the last meeting of the year, with the LIS Chair.  The LIS Chair then integrates those data and discussions with course evaluation and survey data over the summer, to serve as input to next year’s planning meeting.

Each step of our curriculum assessment process is linked with our Student Learning Outcomes (SLOs) and Program Goals. SLOs 1-4 directly link to Program Goal 1 (Provide a curriculum that meets the evolving demands of the job market for librarians and other information professionals), and SLO 5 links with Program Goal 5 (Strengthen the emphasis on Hawaiʻi and the Asia-Pacific region in teaching, research, and service). Every course is aligned with the particular SLOs it covers via a matrix.

For the culminating experience, the ePortfolio and oral exam scenarios are also linked with our SLOs, and students are evaluated along the same dimensions (exceed-meet-approach-do not meet). We use this data to revise both the curriculum and the oral exams themselves.

Thesis students are assessed on specific elements of their research, including their problem statement, literature review, method, results, and the quality of their written thesis and oral defense; the evaluation form can be found on p. 7 of our thesis policy and FAQ.

Retention and persistence

From Fall 2013-Fall 2018, our student retention was 83% after one year, 74% after two years.

Time to degree

From Fall 2009-Fall 2018, the overall graduation rate has been 89%, with average time to graduation between 2.0 – 2.5 years.

Job placement

In our 2018 Alumni survey, 60 graduates were contacted and 21 responded (35%).  81% of respondents reported that they were working in an LIS-related professional position within 12 months of graduation, primarily in academic (29%), public (29%), archives (14%), special (14%) or school (10%) libraries.  Most alumni (62%) work in Hawaii, with 24% working on the US mainland.

Placement information for our 43 school library media specialist graduates since 2009 is reported in our 2015 ALA Accreditation Program Presentation (Appendix 2-17, Practicum Report, section E).

In the 2017 Library Journal survey, we sent a request to the 29 alumni (26 women and 3 men) who graduated during the prior year. Sixteen of the 29 students completed the survey (55.2%), which was the ninth highest rate among reporting schools. Of the 16 respondents, 11 were employed full-time, and reported salaries ranging between $41, 244 and $53,510, with an average of $45,439.

Additional data is also available via the Library Journal Placement & Salaries Survey.

Job performance and advancement

We survey employers of our graduates every 5 years, most recently in 2019.  We ask employers to indicate the extent to which UH LIS graduates are able to demonstrate skills in three broad areas, and over 70% of respondents agreed or strongly agreed with all three of these statements of our graduates’ abilities:

  • Provide information services for a range of users’ needs
  • Practice ethical responsibilities of the profession in providing services and resources
  • Manage and work effectively in collaborative problem solving and team projects

Respondents also indicated that UHM LIS graduates demonstrate consistent strengths in the following:

  • Inclusion of indigenous and local knowledge to inform library practices
  • Passion for the field and a deep interest in serving patrons
  • Critical thinking and ability to analyze data
  • Teamwork
  • Reference services and information retrieval
  • Web design and digital preservation
  • Community engagement and customer service skills
  • Connections with professional communities
  • Ability to meet a range of user needs
  • Work ethic
  • Overall professionalism

Complete responses, including job advancement data, can be found in our 2019 Employer Survey Summary.

For the 2013-2014 review period, the Program also requested access to principals’ ratings of LIS graduates on the Hawai‘i DOE’s Educator Effectiveness System (EES). The EES uses a 4-point rating scale and ten of eleven graduates hired since 2009 were rated with the following criteria:

  • Highly effective: met or exceeded at least 90% to 100% of expected target.
  • Effective: met or exceeded at least 75% to 89% of expected target.
  • Developing: met or exceeded at least 60% to 74% of expected target.
  • Ineffective: met less than 60% of expected target.

Nine of the ten librarians received “effective” or “highly effective” ratings on foci they had selected for the evaluation.  Licensure pass rates (as applicable) were 100%.

Program and outcomes assessment

Annual assessment reports submitted to the Manoa Assessment Office, and previously to the Vice Chancellor for Academic Affairs and Graduate Division, are also available for review: