University of Nebraska Kearney

Specialist (Superintendent/District Level) Graduate Program Assessment: Assessment Report 2006 

Submitted Fall, 2006

Department of Educational Administration
Assessment Report for District Level Administrators
September 2006

Assessment Mission Statement

Graduates will be competent district level school administrators, who are responsive, reflective, and collaborative decision makers and who have the requisite knowledge, skills, and technological expertise to organize learning environments that embrace the principles of democracy, diversity, and equity.

Assessment Goal:

Graduates will become competent school administrators.

A. Superintendent/District Level Objectives 

Knowledge:

Learning Objective 1:
Candidates who complete the program are educational leaders who have the knowledge and ability to promote the success of all students by facilitating the development, articulation, implementation, and stewardship of a vision of learning that is shared and supported by the school district.

Learning Objective 2:
Candidates who complete the program are educational leaders who have the knowledge and ability to promote the success of all students by promoting a positive school culture, providing an effective instructional program, applying best practices to student learning, and designing comprehensive professional growth plans for staff.

Learning Objective 3:
Candidates who complete the program are educational leaders who have the knowledge and ability to promote the success of all students by understanding, responding to, and influencing the larger political, social, economic, legal, and cultural index.

Learning Objective 4:
Candidates who complete the program are educational leaders who have the knowledge and ability to promote the success of all students by managing the organization, operations, and resources in a way that promotes a safe, efficient, and effective learning environment

Skills:
Learning Objective 5:
Candidates who complete the program are educational leaders who communicate effectively both orally and in writing.

Learning Objective 6:
Candidates who complete the program are educational leaders who are able to design and conduct research, analyze data, and make recommendations.

Learning Objective 7:
Internship. The internship will provide a significant opportunity for candidates to synthesize and apply the knowledge and practice and develop the skills identified in state and local standards through substantial, sustained, standards-based work in real settings, planned and guided cooperatively by the institution and school district personnel for graduate credit.

Values:
Learning Objective 8:
Candidates who complete the program are educational leaders who have the knowledge and ability to promote the success of all students by acting with integrity, fairly, and in an ethical manner.

Learning Objective 9:
Candidates who complete the program are educational leaders who have the knowledge and ability to promote the success of all students by collaborating with families and other community members, responding to diverse community interests and needs, and mobilizing community resources.

Learning Objective 10:
Candidates who complete the program are educational leaders who seek to increase their education by formal and informal means.

2006 Assessment Report

Both direct and indirect measures were used for assessment, including portfolio assessments, parallel surveys for employers and alumni, Advisory Group Surveys, and a database on graduate placements.

Direct:
D1 Oral Assessment (Mock Interview)
D2. Internship Assessment
D3. Field Study Assessment
D4. Field Study Oral Defense

Indirect:
Ind. 1 Alumni Parallel Survey
Ind. 2 Employer Parallel Survey
Ind. 3 Advisory Committee Survey
Ind. 4 Placement (Database)

Assessment Interpretation and Recommendations:

Direct Assessment 1: Oral Assessment [Mock Interview (Objectives 1,2,3,4,5,8,9)]

Assessment

Student learning outcomes were measured with an oral assessment in the form of a mock interview. Topics addressed were public relations, school law, school facilities, business management, and administrative theory, the framework for understanding, predicting, and controlling performances in their school system. All faculty members used a prepared script based on a format from Ventures in Excellence, a research based company. As a team, the members of the department discussed and assessed the responses based on a data bank of compiled responses that are indicators of successful administrators. Prior to 2003, the assessment was based on pass or fail. During the spring and summer of 2003, assessment items were keyed to specific objectives. Each student received an Overall Assessment of Basic, Capable or Proficient.

During the fall 2005, spring 2006, and summer of 2006, seven students participated in the Mock Interview. All students passed. The performance assessment allowed for immediate feedback to the candidate and faculty members as to whether the learning objectives were being met and if there were a need for program changes. Department members were in agreement that the Mock Interview measured the designated learning objectives.

Recommendations

Data indicate that our current program is meeting the department goals and objectives included under this assessment, however since the numbers were small it was suggested to complie as one graph whenever the student numbers are under 10.

Direct Assessment 2: Internship Assessment (Objectives 1,2,3,4,5,7,8,9)

Assessment

All candidates participated in an Internship, an experiential assignment, which included activities with substantial responsibilities that increased over time in amount and complexity and involved direct interaction and involvement with staff, students, parents, and community leaders. Students maintained logs of their internship experiences. The site mentors completed a survey evaluating the performance of the six students who interned in their school during spring and summer 2005 and completed their internships. Students were assessed on a scale from 1-Outstanding to 5-Inadequate with a NO for No Opportunity to Observe. When completed, the site mentor returned the survey to the university supervisor. All students received satisfactory ratings on logs and surveys. The findings indicate that our department goals and objectives are being met by our department offerings.

Recommendation:

Data indicate that our current program is meeting the department goals and objectives included under this assessment.

Direct Assessment 3: Field Study Assessment (Objectives 6)

Assessment

The Field Study is a written project designed to provide the candidates in educational administration with the opportunity to design and conduct research on a contemporary issue in education. Each Field Study is evaluated as pass or fail by a Field Study Committee composed of the advisor of field studies, a College of Education (COE) faculty member who teaches Techniques of Research, one other faculty member from EDAD; and a graduate faculty member at large from the University of Nebraska faculty or an additional graduate faculty member from the candidate’s department.

Eight students completed field studies. Assessment was based on design, statistical application, literature review, and interpretation of findings. All students successfully completed their field studies.

Recommendations

Data indicated that our current program is meeting the department goals and objectives.

Direct Assessment 4: Oral Defense of Field Study (Objective 5 and 6)

Assessment

After completion of the Field Study, the candidate defended his or her research before their Field Study Committee. The assessment allowed for immediate feedback to the students.

All eight students successfully completed their oral defenses of their field studies. Currently, the three areas addressed—explanation of findings, related statistical measures, and recommendations—are assessed as pass or fail.

Recommendation

Data indicated that our current program is meeting the department goals and objectives.

Indirect Assessment 1 and 2: Surveys (Objectives 1,2,3,4,8,9,10)

Assessment

To provide insights into the levels of preparedness graduates experience when becoming administrators and the strengths and weaknesses of the EDAD program in contributing to their readiness, parallel surveys were sent to employers and alumni. Every three years parallel surveys are sent to a random sampling of UNK EDAD Program graduates who are currently practicing school administrators in Nebraska and to their Board Presidents. Results of the survey were compiled and examined to determine if there were implications for changes in EDAD programs. Information was shared with both inter-and external audience, and for making recommendations and, if needed, program changes.

On April 1, 2005 the University of Nebraska at Kearny’s Educational Administration Department (UNK EDAD) sent out its triennial graduate survey. Completed questionnaires were received from 58 of 95 Master degree graduates for a return rate of 61%. Completed surveys were returned from 27 of 76 Specialist degree graduates for a return rate of 36%.

Two additional components were added to the 2005 survey. Superintendents who supervised recent graduates of the UNK EDAD Masters program were asked to rate those administrators using the same areas as the self-rating questionnaire sent to the principals. Board Presidents who were on boards of education that employed recent graduates of UNK EDAD Specialist program were asked to rate those superintendents using the same areas as the self-rating questionnaire sent to the superintendents. Completed questionnaires were received from 27 of 76 superintendents for a return rate of 36%. Completed surveys were returned from 20 of 76 board presidents for a return rate of 26%.

All 2005 survey information was provided in the 2005 Assessment Report. The next survey will be conducted in 2008.
 

Indirect Assessment 3: Advisory Committee Evaluation (Objectives 8, 9, 10)

Assessment

To provide statewide insights into the perceived quality of our UNK administrators, alumni and members of the Nebraska Department of Education (NDE) were invited to serve on the EDAD Advisory Committee. An agenda was mailed prior to the meeting and time was allocated for concerns or comments. At the annual meeting, committee members were requested to react to a set of questions that focused on their perceptions of UNK graduates. Responses were compiled, analyzed, and evaluated. The members of the committee rated our graduates high in all areas.

Eight committee members completed assessments. The following questions were posed: (1) Graduates have the knowledge and ability to promote the success of all students.
(2) The graduates are sought as candidates for administrative positions. (3) State educators are both aware and have respect for the EDAD graduate. Based on agree or disagree, all committee members responded, agree, on all questions. On questions four, Summative Evaluation: How would you rank the EDAD graduates at the University of Nebraska at Kearney: exemplary--3; excellent—4; and one who couldn’t decide, marked both Exemplary and Excellence.

Each question contained a place for Strengths and Concerns and Suggestions. Questions one and five received most comments. To better make program improvement, we focused on the Concerns and Suggestions.

Question 1:

1. Continue to determine the best ways to balance the need for face-to-face contacts with online courses.
2. Perhaps, use local mentoring, especially for those taking courses online.
3. I like the idea of “passing the torch” by having practicing administrators identify candidates.
4. In the dark ages when I went through the program, I learned as much from students (and their experience in their schools) as I did from instructors. My concern is that efforts need to be made to generate that [student interaction] and it is difficult when not face-to-face.

Question 5:

1. Broaden perspective to national and international without losing local/state understanding.
2. Continue broadening delivery systems as technology becomes available.
3. More face-to-face opportunities interspersed with online instruction.
4. Get the students together earlier in the course works, especially if the course is delivered via internet.
5. Continue to solicit and listen to current practitioners.
6. Either a mentor concept or an emphasis on assignments that regularly require students to access direct input from experienced administrators.
7. Local mentor recruitment for all EdAd students
8. Be visible and involved in off-campus conversations about leadership in Nebraska Schools ( High school reform efforts, NDE regional and state meetings)
9. Put together an aggressive “on site” recruitment programs.

Recommendation:

The findings indicate that our department goals and objectives are being met by our department offerings; however, based on the comments, the department needs to continue to seek innovative technology to create a face-to-face environment in online classes and to continue to maintain the partnership between public school administrators and educational administration students.

Indirect Assessment 4: Placement (Objective 8)

Assessment

In addition to program-specific assessments, another measurement was included to assess program outcome effectiveness. Job placements of candidates were recorded in a database to document program effectiveness and success for all students.

A map depicting EDAD Graduate Employment by County, 1999- 2006, can be found at the end of the report.

Recommendation

The findings indicate that our department goals and objectives are being met by our department offerings.

Assessment of the Assessment Process

The faculty members have considered the following:

1. Is the data collected relevant to the department objectives?

   The department objectives are aligned with the College of Education (COE) Ten Desired Outcomes; COE Dispositions; COE Conceptual Framework; Nebraska Department of Education (NDE) Rule #24; and the National Policy Board for Educational Administration (NPBEA) Standards.  The assessments are aligned with the objectives; therefore the faculty felt that the data collected was relevant to the department objectives. 

2. Is the focus of the assessment based on “best practices”?

   In the Instruction to Implement Standards for Advanced Programs In Educational Leadership, it is noted: “Tomorrow’s educational leaders must be able to work with diverse groups and to integrate ideas to solve a continuous flow of problems.” The NPBEA standards provide a guide to ensure that our graduates can accomplish tasks associated with each standard and to participate in an extensive internship. 

3. Is the assessment data being used for department decision making?

   At the 2006 spring Advisory Council meeting, a concern was expressed about maintaining a face-to-face environment in an online setting.  Suggested was to maintain a close relationship with area practitioners.
   
   This fall, several department members added Audacity and the web cam to better interact with students.  With the use of Audacity, the students are able to not only read class notes and comments, but are able to hear them also.  With the use of the web cam, another dimension is added.  Students are able to see their instructors.  Department faculty members are continually moving out of their comfort zone to use current technology to create an environment of community. 

4. Are the various strategies for assessing student learning outcomes successful?

   The faculty members feel that there is an appropriate mix of direct and indirect assessment strategies. 

5. Is there a need to discontinue or add any assessment activities?

   Starting after the 2002 Triennial Graduate Survey, there had been a gradual move to offer the Educational Specialist Program online.  Currently, the Educational Specialist Program in online. 
   
   Overall, department members believe that our assessment practices are relevant, provide excellent feedback for the evaluation of our program and present direction for program changes.