Submitted Fall, 2006
Department of Teacher Education
Assessment Report for the Master of Science in Education
Instructional Technology Program
2005-2006 Academic Year
Graduate Programs Statement
Master degree students graduating from the Department of Teacher Education's graduate programs will demonstrate an understanding of the manner in which perspectives and beliefs influence instructional behaviors, extend their knowledge of the central concepts of selected disciplines, and obtain and use knowledge and skills which will support increased instructional effectiveness with diverse learners.
The Department's Graduate Program Objectives
1) Students will demonstrate their leadership and administrative skills, which include promoting and implementing professional and state standards;
2) Students will effectively design instruction and materials for all learners;
3) Students will demonstrate how to use a variety of productivity, research, and software tools in various instructional settings;
4a) Students will be able to apply, analyze, synthesize and evaluate critical and contemporary issues; and,
4b) Students will be knowledgeable consumers and generators of research
Instructional Technology Objectives
1) Students will demonstrate their leadership and administrative skills, which include promoting and implementing the International Society for Technology in Education (ISTE) standards in Instructional Technology or in Educational Media; (Department Objective 1)
2) Students will design instructional modules which provide content to their students and meet the ISTE NETS and/or ALA Media Standards; (Department Objective 2)
3) Using a variety of software programs, students will design, develop, create, and implement various instructional modules for their classrooms and/or training environments. (Department Objective 3)
4) Students will investigate or conduct current research into various contemporary issues in their area and will demonstrate their analysis of the issue to their colleagues, school boards, school administration, parents, or similar stakeholder groups. (Department Objectives 4a, 4b)
Data Report
As per the approved assessment plan for North Central Accreditation, during the 2005-2006 Academic Year, which includes the summer of 2006, 48 graduates of the IT program completed direct assessment measure activities, while another 26 completed an indirect measure activity.
The direct measure activity completed by our graduates was the completion and submission of an electronic portfolio which included reflective statements and exemplars of their work. Students were rated on each objective using our approved portfolio rubric. Each component of the portfolio was scored as unacceptable, acceptable or commendable. Students were allowed to rework unacceptable artifacts and reflective statements and resubmit them. Students also worked with their advisor and received formative feedback during the creation process. Many of the students also took a class in portfolio development in which they began the process of portfolio creation.
The indirect measure, which was the Alumni Survey, was sent to 48 graduates of the program. Approximately 7 were returned by the US mail service for lack of valid addresses. 26 completed surveys were returned. It was originally decided that we would wait for at least two years before we surveyed our graduates in order to give them time to see what skills they might be lacking. However, that time period also allowed us to lose contact with many of them. The form was rewritten and was (and will continue to be) administered immediately after graduation.
Direct Assessment
Portfolio Assessment for Instructional Technology
| Program Objective 1: Students will demonstrate their leadership and administrative skills, which include promoting and implementing the International Society for Technology in Education (ISTE) standards in Instructional Technology or in Educational Media |
| Students will |
Unacceptable |
Acceptable |
Commendable |
| demonstrate the use of technology for teaching and learning |
Little or no evidence that the student can use technology for teaching and learning
|
Clear evidence that the student can adequately use appropriate technology for teaching and learning
|
Clear and consistent evidence that the student can integrate appropriate uses of technology for teaching and learning
|
| Student scores for this enabling objective |
0% |
58% |
42% |
| demonstrate the use of technology in regards to legal and ethical issues, including copyright, privacy, and security of technology systems, data, and information |
Little or no evidence that the student can use technology in regards to legal and ethical issues, including copyright, privacy, and security of technology systems, data, and information
|
Clear evidence that the student can adequately use technology in regards to legal and ethical issues, including copyright, privacy, and security of technology systems, data, and information
|
Clear and consistent evidence that the student can evaluate technology-related usage in regards to legal and ethical issues, including copyright, privacy, and security of technology systems, data, and information
|
| Student scores for this enabling objective |
0% |
54% |
46% |
| Program Objective 2: Students will design instructional modules which provide content to their students and meet the ISTE NETS and/or ALA Media Standards |
| Students will |
Unacceptable |
Acceptable |
Commendable |
| demonstrate the use of technology to maximize student learning and facilitate higher order thinking skills |
Little or no evidence that the student can use technology to maximize student learning and facilitate higher order thinking skills
|
Clear evidence that the student can adequately use technology to maximize student learning and facilitate higher order thinking skills
|
Clear and consistent evidence that the student can use technology to maximize student learning and facilitate higher order thinking skills
|
| Student scores for this enabling objective |
0% |
71% |
29% |
| demonstrate that they can design and teach technology-enriched learning activities that connect content standards with student technology standards and meet the diverse needs of students. |
Little or no evidence that the student can design and teach technology-enriched learning activities that do not connect content standards with student technology standards nor meet the diverse needs of students.
|
Clear evidence that the student can design and teach technology-enriched learning activities that connect content standards with student technology standards but do not meet the diverse needs of students.
|
Clear and consistent evidence that the student can design and teach technology-enriched learning activities that connect content standards with student technology standards and meet the diverse needs of students.
|
| Student scores for this enabling objective |
0% |
48% |
52% |
| Program Objective 3: Using a variety of software programs, students will design, develop, create, and implement various instructional modules for their classrooms and/or training environments |
| Students will |
Unacceptable |
Acceptable |
Commendable |
| demonstrate the use of technology-based assessment strategies and tools for evaluating specific learning activities |
Little or no evidence that the student can use technology-based assessment strategies and tools for evaluating specific learning activities
|
Clear evidence that the student can adequately use technology-based assessment strategies and tools for evaluating specific learning activities
|
Clear and consistent evidence that the student can integrate technology-based assessment strategies and tools into plans for evaluating specific learning activities
|
| Student scores for this enabling objective |
0% |
63% |
37% |
| Demonstrate the use of specific technology applications and resources that maximize student learning, address learner needs, and affirm diversity |
Little or no evidence that the student can use specific technology applications and resources that maximize student learning, address learner needs, and affirm diversity
|
Clear evidence that the student can adequately use specific technology applications and resources that maximize student learning, address learner needs, and affirm diversity
|
Clear and consistent evidence that the student can integrate specific technology applications and resources that maximize student learning, address learner needs, and affirm diversity
|
| Student scores for this enabling objective |
0% |
52% |
48% |
| Program Objective 4: Students will investigate or conduct current research into various contemporary issues in their area and will demonstrate their analysis of the issue to their colleagues, school boards, school administration, parents, or similar stakeholder groups |
| Students will |
Unacceptable |
Acceptable |
Commendable |
| demonstrate understanding of research and can evaluate the accuracy, relevance, appropriateness, comprehensiveness, and bias of electronic information resources. |
Little or no evidence that the student can evaluate the accuracy, relevance, appropriateness, comprehensiveness, and bias of electronic information resources.
|
Clear evidence that the student understands research and can evaluate the accuracy, relevance, appropriateness, comprehensiveness, and bias of electronic information resources.
|
Clear and consistent evidence that the student can conduct research and evaluate the accuracy, relevance, appropriateness, comprehensiveness, and bias of electronic information resources.
|
| Student scores for this enabling objective |
0% |
42% |
58% |
| identify, select, and use hardware and software technology resources specially designed for use by PK-12 students and others to meet specific teaching and learning objective |
Little or no evidence that the student can identify, select, and use hardware and software technology resources specially designed for use by PK-12 students and others to meet specific teaching and learning objectives
|
Clear evidence that the student can identify, select, and use some hardware and some software technology resources specially designed for use by PK-12 students and others to meet specific teaching and learning objectives
|
Clear and consistent evidence that the student can identify, select, and use hardware and software technology resources specially designed for use by PK-12 students and others to meet specific teaching and learning objectives
|
| Student scores for this enabling objective |
0% |
19% |
81% |
| |
| Average scores for all objectives |
0% |
50.88% |
49.13% |
Indirect Assessment
As cited above, 26 Alumni surveys were returned. The responses were based on the following responses:
| Strongly agree |
Agree |
Neither agree
nor disagree |
Disagree |
Strongly disagree |
Not applicable |
| 5 |
4 |
3 |
2 |
1 |
0 |
- I have the ability to promote and implement the International Society for Technology in Education (ISTE) standards in Instructional Technology or in Educational Media;
Mean response 4.48
- I have the ability design instructional modules which provide content to my students.
Mean response 4.38
- I have the ability to design instructional modules which meet the ISTE NETS and/or ALA Media Standards
Mean response 4.63
- I am able to use a variety of software programs to design various instructional modules for my classrooms or training environments.
Mean response 4.52
- I am able to use a variety of software programs to develop various instructional modules for my classrooms or training environments.
Mean response 4.54
- I am able to use a variety of software programs to create various instructional modules for my classrooms or training environments.
Mean response 4.72
- I am able to use a variety of software programs to implement various instructional modules for my classrooms or training environments.
Mean response 4.68
- I am able to investigate or conduct current research into various contemporary issues in my area.
Mean response 4.64
- I am able to demonstrate my research analysis of the issues to my colleagues, school boards, school administration, parents, or similar stakeholder groups
Mean response 4.36
I found the Graduation Portfolio useful and/or valuable
Mean response 4.85
Mean of all of the responses to the questions was 4.58, an increase of .42 points from 2005
Analysis of Graduate Survey
The overall response rate was much better than last year, but was still to low in our estimation. We received completed instruments from 26 out of 48 graduates. This compares to 17 out of 119 from last year (which included multiple years) or 54.16% compared to only 14.28% last year of the graduates. We expect to have the instrument online next year and believe we will return even a higher rate
The scores last year indicted that we had apparently weaknesses in two areas in the program. One was the inclusion, or specifically the lack of discussion, about the inclusion of ISTE Standards and the other is the research class. Although students met or exceeded the Standards, they apparently did not realize that they did; the Standards were not really discussed in classes except for occasional comments. This year there was emphasis placed on noting the relationship between the program and the Standards. The mean score of the responses increase from 3.29 to 4.48, a 1.19 point increase. Our plan to focus more on the relationship apparently worked
The other low scoring area was the research class. Most of the 2005 respondents, due to when they graduated, were required to come to come to campus to take the research class or find a graduate level research class in their area. The specific on-campus class, which is an excellent research class, is not particularly well-suited for students who do not continue on and complete a field project or a thesis and it was not an online class. These two issues have been addressed by an online research class (TE 800) that has a different focus. Last year we summarized this section by stating "It will be interesting to see how the students respond to those questions in future surveys." The mean score for Question 8 increased from 3.18 to 4.68 and for Question 9, the increase was from 3.13 to 4.64, a combined increase of 1.5.
Although both areas saw substantial improvement, our students still scored the Standards question (#3) and the Research question (#9) the lowest of the 10 questions.
Critical Friends Survey
The Critical Friends Survey was mailed to 25 individuals in various Education Service Units (ESU), principals and superintendents, and technology coordinators around the state. At this time, no out of state Critical Friends have been identified. Only seven were returned and most of those had numerous "no response" selected (left blank). This is very disconcerting. The following are the responses.
Indirect Assessment
As cited above, seven Critical Friends surveys were returned. The responses were based on the following responses:
| Strongly agree |
Agree |
Neither agree
nor disagree |
Disagree |
Strongly disagree |
Not applicable |
| 5 |
4 |
3 |
2 |
1 |
0 |
- Your graduates have the ability to promote and implement the International Society for Technology in Education (ISTE) standards in Instructional Technology or in Educational Media;
Mean response 4.00
- Your graduates have the ability design instructional modules which provide content to their students.
Mean response 4.17
- Your graduates have the ability to design instructional modules which meet the ISTE NETS and/or ALA Media Standards
Mean response 3.33
- Your graduates are able to use a variety of software programs to design various instructional modules for their classrooms or training environments.
Mean response 4.25
- Your graduates are able to use a variety of software programs to develop various instructional modules for their classrooms or training environments.
Mean response 4.00
- Your graduates are able to use a variety of software programs to create various instructional modules for their classrooms or training environments.
Mean response 4.25
- Your graduates are able to use a variety of software programs to implement various instructional modules for their classrooms or training environments.
Mean response 4.25
- Your graduates are able to investigate or conduct current research into various contemporary issues in their area.
Mean response 4.00
- Your graduates are able to demonstrate their research analysis of the issues to their colleagues, school boards, school administration, parents, or similar stakeholder groups
Mean response 4.00
Mean of all of the responses to the questions was 4.03
Analysis of the Critical Friends Survey
It is difficult to draw conclusions from the survey with less than 30% of the Friends returning the instrument. In addition, of the ones that did return the surveys, many respondents only answered a question of two. It is also apparent that the Friends did not have much of an understanding of what we were even asking. If the ignored questions were computed into the total mean, it would be 2.51.Something must be done to improve the response rate.
Recommendations
Both the Alumni Survey and the Critical Friends Survey need to be offered online. Then the URL ca be sent to the individuals either as a paper copy of , preferred, in an email that they can click on the link and go to it. This should increased the return rate.
An informational document should be developed to provide the Critical Friends group with information about ISTE, the program, etc. of which they might be unaware. In addition, we might consider changing the nature of the Critical Friends to alumni who have graduated five or more years ago.
Although the return rate for the alumni was fair to middling, it could be improved by obtaining a “real” email address for our graduates and not just use the UNK, particularly when the survey gets online.