The unit has an assessment system that collects and analyzes data on the applicant qualifications, the candidate and graduate performance, and unit operations to evaluate and improve the unit and its programs.

The COE assessment system collects, organizes, and analyzes data about applicant qualifications, candidate and graduate performance, and unit operations. These data: (a) allow formative assessment of candidates at key transition points in their programs; (b) provide summative judgments for institutional recommendations of graduates; (c) allow formative assessment to strengthen initial and advanced preparation programs; (d) contribute to faculty members’, administrators’, and school partners’ ability to improve practice; and (e) inform unit decision-making with a focus on improvement. (See Data Collection Schedule).

Chronological Development of Unit Assessment System and Alignment with Conceptual Framework
The original assessment plan was developed during the 1999-2001 academic years, in conjunction with the Long Range Plan, which specified the college would monitor and evaluate its ongoing operations, scope of work, quality of offerings, and graduates' effectiveness. The Conceptual Framework and state and national standards identified in the Professional Standards Matrix drive each aspect of the assessment system. The matrix illustrates alignment of candidate proficiencies with the conceptual framework as well as institutional, state, and national standards.

2.1 Assessment System

Figure 2.1 Components of Assessment System

Candidate Data
The core of the assessment system is evaluation of candidate performance using multiple assessments from internal and external sources at key transition points in each initial and advanced program. These transition points, depending on the program, may include entry, early, middle, late, exit, and follow-up assessments.

Initial Teacher Preparation
Entry-level assessments for teacher education candidates include program application, PPST (PRAXIS I) scores, Content (PRAXIS II) scores, admission essay and interview to identify professional dispositions, clinical field experiences, as well as cumulative and content grade point averages. Early, middle, late, and exit level assessments may include candidates' grades in required and elective courses, self-assessment, mentor teacher and faculty assessment, projects, work samples, electronic portfolios, and dispositions ratings. Program rubrics and other criteria determine candidates' accomplishments. Exit level assessment also includes candidates' performance on state licensure examinations.

Teacher education candidates develop an electronic portfolio that includes course products and work samples, which are submitted and reviewed at various points in their preparation programs. The College has 15 Teacher Education Committees (TECs) comprised of COE and Arts and Sciences faculty, students, HI DOE representatives, and other personnel who make program recommendations regarding curricula, field experiences, partnerships, and other important issues.

Follow–up evaluation is also an important component of the assessment system. Therefore, the College implements systematic collection of survey data every three years from alumni, employers, and mentor teachers as well as current masters and doctoral degree candidates. In addition, each semester candidates in student teaching and internship settings evaluate their program experiences. Facilitated focus groups and an appreciation conference with mentor teachers also provide feedback data.

Advanced Preparation Programs
Assessment of candidate proficiencies in advanced programs is not based on a common set of evaluation instruments, but rather is program specific. The conceptual framework is the common assessment element in these programs.

Candidates who express interest in advanced programs are assigned an advisor who discusses program requirements and monitors individual progress to ensure completion in a timely manner. Systematic data collection at specific transition points during candidates' enrollment support continuation in the program. Advanced program candidates must meet university GPA, degree, and entrance examination requirements. Some advanced programs require standardized test scores (e.g. GRE, TOEFL), writing samples, and interviews as well as practicum or internship, which candidates take on an elective basis that provide performance assessment data.

The information in Table 2.1 delineates the key transition points at which the College conducts assessments of candidates enrolled in initial and advanced preparation programs. The documents, Unit Assessment System Initial Programs and Unit Assessment System Advanced Programs, delineate specific assessments and data collection at key transition points in each initial and advanced program (see Table 2.1 Unit Assessment System: Data Collection at Key Transition Points and Table 2.2 Other Candidate Data).

The assessment system allows the college to monitor progress toward attaining its vision and mission, as explicated in the conceptual framework, as well as provide support for programmatic decisions related to preparing educators who demonstrate its core values. Assessment measures are drawn from content, professional, and pedagogical knowledge, skills, and dispositions discussed in Standard 1 and ensure that candidates meet standards as they advance through programs.

Opportunities for Remediation
The college tries to ensure that program expectations and requirements are clear to all candidates. University print and online publications detail program requirements. In addition, faculty and staff are available to advise students and clarify any points of concern. However, at times, candidates may not perform to expectations. Candidates may be referred to UHM advising and counseling services, or the UHM reading and writing center depending on the difficulty. Individual faculty members may provide remediation plans and the opportunity to re-take assessments or tests, or they may design individualized interventions.

Despite these efforts, candidates occasionally are unable to meet requirements at the key transition points in a timely fashion. They may be: (a) placed on academic probation; (b) denied advancement in their preparation programs; (c) asked to follow a plan of assistance; and/or (d) encouraged to explore career alternatives other than the field of education. The Assistant Dean for Student Academic Services maintains records of formal candidate complaints, investigations, and resolutions.

Test of Fairness, Accuracy, and Consistency and Avoidance of Bias
Assurances of fairness, accuracy, consistency, and avoiding bias in measuring candidate performance are ongoing challenges and vary with the program and assessment instruments. (See Initial Program Assessment Initiatives and Advanced Program Assessment Initiatives for specific considerations regarding fairness, accuracy, consistency, and avoiding bias). Therefore, the College relies on multiple internal and external measures to guarantee these undertakings. Measures include, but are not limited to the following:

Fairness and Accuracy
• Content of the assessment instrument comparison with course syllabi, or expectations for candidates' performance in clinical settings.
• Common syllabi and assessment of courses and clinical experiences with rubrics to measure candidates on the same knowledge and skills regardless of who teaches the class, and to ensure that expectations for candidates are clear and unambiguous.
• Standardized format for course syllabi, in which faculty members indicate performance expectations and grading measures, standards addressed by the course, and other information to ensure clarity of course requirements and operations.
• Aggregated data reporting to reveal differences among faculty, programs, or student groups.
• Opportunities for candidates to remediate gaps in knowledge and skills.
• Campus services for candidate assistance and accommodations for those with disabilities (e.g., reading and math labs, writing center, advising center, disability and non-traditional student programs).
• Statements on course syllabi that encourage candidates with disabilities to seek accommodations, as necessary.
• Appeals process for candidates who disagree with assessment(s).

Consistency
• Standardized rubrics for assessment instruments, observations, and portfolios to ensure consistency across sections of courses and evaluation of field experiences.
• Common grading processes (rubrics) for comprehensive examinations to create shared faculty expectations and judgments.
• Faculty discussions to standardize expectations for candidates' performance at various transition points and compare these expectations with assessment instruments.
• Discussions with cooperating/mentor teachers and university supervisors that address fairness, accuracy, consistency, and non-biased evaluation of candidates.
• Examinations of inter-rater reliability of measures of candidate performance in clinical experiences and assessment of portfolios.
• Handbooks for teacher education and advanced programs designed in a common format and which include statements regarding non-discrimination.

Multiple measures of candidates at critical transition points also help to guarantee fairness, accuracy, consistency, and avoidance of bias. These include:
• Entrance criteria;
• Standardized and performance based evaluation of candidates’ course and field experiences;
• Comparative data analysis (COE, and school site data) of candidate’s performance in P-12 settings during field/clinical experiences;
• Course assignments with rubrics that assess candidates’ content and pedagogical knowledge, skills, and dispositions and reflection about their knowledge, skills, and dispositions;
• Candidates’ perceptions of their preparation to perform the standards of their profession effectively upon exiting the program and after one to two years in their field;
• Exit criteria;
• Employers’ evaluation of graduate performance;
• Comparative data analysis of graduates’ and employers’ assessment of graduates’ performance.

Multiple assessment sources for the unit and programs, which help to guarantee fairness, accuracy, consistency, and avoidance of bias include:
• Candidates’ perceptions about unit, program, and university services and resources;
• Candidates’ evaluation of faculty;
• Candidates, faculty, and staff perceptions about the unit and program effectiveness in dealing with issues of diversity and technology;
• Faculty and staff perceptions about the Conceptual Framework and their respective roles and responsibilities as well as perceptions about resources and professional development;
• Individual program self-study, assessment plans, and annual reports;
• Input form advisory boards, faculty and staff retreats, and various committees.

Faculty, Resource and Productivity Data
The above discussion considers how student data help assess individual candidate’s progress and readiness for an institutional recommendation. Discussion in Element 2.3 addresses how data contribute to program improvement efforts.

Faculty Data
Faculty data that are part of the unit assessment system include: gender, ethnicity, rank and tenure status, retention and promotion documentation, research, scholarly activity, and professional service. Candidates complete evaluations of each course using a department-wide course and faculty assessment instrument. This process provides instructors with a summary of how candidates rate course organization and instructional quality, as well as value and difficulty of the course. See Standard 5 for a detailed discussion of course evaluation(s).

Resource Data
The assessment system considers faculty and staff productivity data, fiscal information, and other resource data, such as full-time faculty equivalents, head counts and demographics, and the number and demographics of staff members who provide administrative and technical support to faculty and candidates. Budget data are also included in the resource section of the assessment system, and include the dollars allocated for equipment, supplies, travel, library acquisitions, technology, graduate assistants, and other costs. See Standard 5 and Standard 6 for detailed discussions of college personnel and fiscal resources.

Productivity Data
This section of the assessment system measures academic efficiency and effectiveness, including information on enrollment for the regular academic year and summer session, numbers of graduates, and student credit hour production. The assessment system also contains data about faculty workloads and assignments, including effort devoted to teaching, research, and service. The data described above are used in a wide variety of ways to inform: (a) candidate progress through their preparation programs; and (b) faculty, program, and college improvement efforts.

Use of Data for Unit Operations
The College relies on assessment of candidates, data from programs, student teaching, and internships, recent graduates, faculty, and other members of the professional community to evaluate its overall operation. Admissions and advising processes, record keeping, and governance structures are assessed and modified as necessary to improve unit operation. The College reviews the following to determine satisfaction with unit operations: admissions and advising processes and data, follow-up data from program graduates and employers, candidate performance in field experiences, candidate evaluation of instruction and field experiences, including supervision, candidate complaints and due process appeals, faculty and administration evaluation, tenure and promotion policies, budget allocations, internal and external program reviews by departments, university bodies, NCATE, and other professional organizations.

2.2 Data Collection, Analysis, and Evaluation

Figure 2.2 shows the entire Assessment Cycle, including data collection, analysis, reporting, actions, and the information management system.

Data Collection
Data are collected at different points throughout the year. Of the more than one dozen candidate data surveys and evaluations, most are collected bi and tri-annually. Each of the five faculty and fiscal reports are collected annually.

Database Maintenance
The College relies on the TK20 system, which is an online, customizable, assessment, accountability and management scheme that offers a complement of tools to support: (a) assessment and accountability; (b) field experiences and practical training; (c) student advisement; (d) mailing and reporting surveys; (e) building and viewing reports; (f) management of documents; (g) communication between users; (h) communication with graduates and alumni; (i) system security; (j) college admissions; and (k) other functionality that allows operational improvements. The database specifies measures at each transition point that reflect candidates’ acquisition of knowledge, skills, and dispositions. It also allows a holistic perspective of the unit and enables faculty to identify program strengths, areas in need of improvement, additional assessments, as well as competency areas and standards that should be strengthened.

Candidate Data
Candidates enter the database when they enroll in a COE program or course. Information about candidates enrolled in a College course is exported from the Banner system to the TK20 database, providing up-to-date demographic and academic information. Faculty members responsible for evaluation of candidates at specified program transition points enter assessment results (e.g., rubric scores, sub-scores, and qualitative outcomes) in the database as they complete the assessments. Similar information for candidates enrolled in programs, which do not utilize the TK20 database, resides in the respective department and program databases and files.

Data on Program Graduates
During the 1997 academic year, the college began annual survey follow-up for all teacher education and their employers. Surveys are designed to ascertain graduates’ satisfaction with program instruction, field experiences, internships, advising, mentoring, faculty support, and relevance to and preparedness for their positions.

Faculty, Productivity and Resource Data
Faculty demographic, salary, and course information enter the TK20 system database from the University wide Peoplesoft information system. These data are transferred to the COE faculty database, merged in the Operational Data Store, and uploaded to the TK20 system in a manner similar to candidate data. Faculty members enter data regarding research and scholarship (e.g., publications and presentations) and service. The dean’s office ensures that data regarding qualifications, rank, tenure, workload agreements, and annual evaluations are current.

Data Analysis
Analyses for Candidate Assessment
Analysis of how programs measure individual candidate competencies also informs program and unit assessment. As shown in Unit Assessment System Initial Programs and Unit Assessment System Advanced Programs, candidate data are gathered and analyzed at key transition points in each program.

Analyses for Faculty Improvement
Faculty workload and activity reports enable the College to standardize and analyze information to determine equitable balances among faculty members’ teaching, research, and service activities. Moreover, the college can assess whether the unit as a whole is fulfilling its obligations across all three areas. Current data show that faculty members engage in instruction, research and service activities. Also, as described earlier, student course evaluations are aggregated for individual courses and instructors, across programs and for the unit as a whole, to allow multiple uses of the data as a basis for identifying areas in need of faculty improvement, and to acknowledge exemplary work. These data, in concert with workload and activity analyses contribute to ensuring a fair faculty review process, where decisions are made on the basis of unbiased information from multiple sources.

Data Analysis for Unit Review
Appropriate measures of success for the unit as a whole include, but are not limited to, student retention and graduation rates across programs, aggregate employer survey results regarding the success of program graduates, and the level of external funding for faculty research. The College also measures success against its mission, asking whether it produces educators who are knowledgeable, effective, and caring.

2.3 Use of Data for Program Improvement and Modification

Faculty members analyze, summarize, and interpret assessment data and use the information to identify: (a) program strengths and weaknesses; (b) trends and foci of curricula and instruction and areas in need of improvement; (c) patterns across program cohorts, and (d) recommendations and actions for program change. As a result of the evaluation, courses may be adapted or developed to address specific needs. In some instances, assignments may be modified across courses to expand candidates’ knowledge.

In addition, until the TK20 assessment system is fully utilized, faculty members use data from a number of sources to guide program development and improvement efforts. Program decisions based on TEC recommendations are forwarded to the Associate Dean for Academic Affairs for action.

Major program modifications such as adding or deleting courses, program admission or exit requirements must have approval of the COE Faculty Senate. Program modifications usually begin in the respective departments, which provide justification for implementing change. The Faculty Senate Curriculum and Program Planning Committee reviews the documents and recommends approval or disapproval to the entire Faculty Senate, which votes on acceptance or rejection of the proposed change. See Initial Program Assessment Initiatives and Advanced Program Assessment Initiatives for specific considerations regarding: (a) use of data to improve candidate and faculty performance and to discuss or initiate change on a regular basis; (b) data driven changes; and (c) information sharing with various constituents.

Summary

The COE assessment system meets the needs of accreditation purposes, but more importantly, supports ongoing evaluation of the unit and its programs. The system reflects the conceptual framework as well as institutional, state, and professional standards.

The system serves three major purposes. The first is assessment of candidates’ performance and ability to demonstrate a positive effect on student learning. The system includes evaluation measures that monitor candidates’ progress at various program transition points.

The second purpose facilitates assessment and improvement of programs that prepare professional education personnel. The system includes multiple internal and external assessment processes, which yield information on the need for program change, and help faculty determine the success of these improvements. Assessment data are shared with candidates, faculty, and other stakeholders as a foundation for improving instructional strategies and practices within public school and/or university classroom settings, modifying the curriculum, and improving field experiences.

The third purpose facilitates assessment of the unit's overall efficiency and effectiveness related to program quality and use of resources. Again, internal and external assessment processes yield information regarding the need for changes within the unit.

TOP