Submitted Fall, 2006
Assessment Report
Department of Communication
September 20, 2006
INTRODUCTION
The data described below were collected through the use of direct and indirect measures in Communication classes, at course presentations, on-line and during portfolio reviews in the fall and spring semesters of the 2005 – 2006 academic year (see Appendix A for direct and indirect measures).
The most recent figures (2006) are contrasted with the data from the previous years throughout this report. In addition to the overall mean for each department goal, the averages from each of the direct and indirect measurement methods used to evaluate that goal are presented.
Regarding the figures below, a 5.00 is the most positive score possible indicating that the evaluators "Strongly Agree" with statements which measure the department’s ability to achieve each of the individual goals. A 1.00 indicates that evaluators “Strongly Disagree” with the statements pertaining to the goals. In terms of interpretation of the data, overall scores that did not meet the 4.00 threshold were earmarked for improvement by the unit’s assessment committee.
More specific information regarding the statement(s) used to evaluate each goal and the numbers employed to produce a mean for each of the six goals can be found in the direct and indirect measuring instruments in Appendix A.
For each of the goals below, averages from the survey of JMC and Speech graduates are not included in the overall mean; note that they are enclosed in parentheses on pages 2 and 3 to indicate this. Approximately 75% of the graduates left the institution before the year 2000 (see Appendix A: Indirect Measures of Graduate Perceptions) and the unit’s programs, goals, staff and instructional practices have necessarily changed a great deal since that time. The belief among the department’s assessment committee members is that if the data from these respondents were included in the calculation of the means for the six department objectives, then the results would be skewed presenting an unrealistic picture of what happened during the 2005 – 2006 academic year. While graduate opinions are always valuable, for the most part their responses do not represent the activities and instruction that took place in the department recently. The quantitative and qualitative data from these measures are useful in that they provide information on the climate of opinion outside of the institution and will be reviewed separately.
SECTION I: DEPARTMENT GOALS
Students will:
- be independent thinkers who are able to develop ideas and clearly express them in oral and written forms;
| |
2006 |
2005 |
2004 |
2003 |
| OVERALL MEAN |
4.28 |
4.17 |
4.42 |
4.24 |
| JMC DIRECT MEASURE |
4.76 |
4.47 |
4.64 |
4.41 |
| JMC PORTFOLIOS DIRECT MEASURE |
4.40 |
4.27 |
4.32 |
4.00 |
| SPCH DIRECT MEASURE |
4.12 |
3.92 |
4.54 |
4.24 |
| JMC INDIRECT MEASURE |
4.21 |
4.03 |
4.20 |
4.13 |
| SPCH INDIRECT MEASURE |
3.92 |
4.14 |
4.23 |
4.40 |
| JMC GRAD. INDIRECT |
(3.85) |
----- |
----- |
----- |
| SPEECH GRAD. INDIRECT |
(3.89) |
----- |
----- |
----- |
- be critical and analytical consumers of messages, regardless of the communication channel;
| |
2006 |
2005 |
2004 |
2003 |
| OVERALL MEAN |
4.00 |
4.10 |
4.22 |
4.00 |
| JMC DIRECT MEASURE |
4.82 |
4.30 |
4.60 |
4.08 |
| JMC PORTFOLIOS DIRECT MEASURE |
4.25 |
4.28 |
4.16 |
3.74 |
| SPCH DIRECT MEASURE |
3.09 |
3.89 |
4.36 |
4.10 |
| JMC INDIRECT MEASURE |
4.04 |
3.98 |
4.17 |
4.10 |
| SPCH INDIRECT MEASURE |
3.82 |
4.07 |
4.07 |
3.97 |
| JMC GRAD. INDIRECT |
(3.60) |
----- |
----- |
----- |
| SPEECH GRAD. INDIRECT |
(3.68) |
----- |
----- |
----- |
- know the history, theory and current issues in the communications discipline;
| |
2006
|
2005 |
2004 |
2003 |
| OVERALL MEAN |
3.99 |
4.00 |
4.17 |
4.03 |
| JMC DIRECT MEASURE |
4.21 |
3.94 |
4.31 |
4.26 |
| JMC PORTFOLIOS DIRECT MEASURE |
4.06 |
4.28 |
4.32 |
3.57 |
| SPCH DIRECT MEASURE |
3.73 |
3.75 |
4.50 |
3.85 |
| JMC INDIRECT MEASURE |
4.09 |
3.99 |
4.12 |
4.03 |
| SPCH INDIRECT MEASURE |
3.84 |
4.06 |
3.92 |
4.09 |
| JMC GRAD. INDIRECT |
(3.82) |
----- |
----- |
----- |
| SPEECH GRAD. INDIRECT |
(3.65) |
----- |
----- |
----- |
- value the freedom of expression;
| |
2006
|
2005 |
2004 |
2003 |
| OVERALL MEAN |
4.15 |
4.06 |
3.79 |
3.85 |
| JMC DIRECT MEASURE |
----- |
----- |
----- |
3.50 |
| JMC PORTFOLIOS DIRECT MEASURE |
----- |
----- |
3.41 |
3.35 |
| SPCH DIRECT MEASURE |
----- |
----- |
3.17 |
3.10 |
| JMC INDIRECT MEASURE |
4.20 |
3.98 |
4.19 |
4.50 |
| SPCH INDIRECT MEASURE |
4.09 |
4.14 |
3.90 |
4.00 |
| JMC GRAD. INDIRECT |
(4.04) |
----- |
----- |
----- |
| SPEECH GRAD. INDIRECT |
(4.15) |
----- |
----- |
----- |
- demonstrate small group skills, including leadership, task completion and relationship development;
| |
2006
|
2005 |
2004 |
2003 |
| OVERALL MEAN |
4.41 |
4.07 |
4.30 |
4.28 |
| JMC DIRECT MEASURE |
4.78 |
4.23 |
4.53 |
4.65 |
| JMC PORTFOLIOS DIRECT MEASURE |
4.44 |
4.39 |
4.50 |
3.92 |
| SPCH DIRECT MEASURE |
5.00 |
3.54 |
4.89 |
3.65 |
| JMC INDIRECT MEASURE |
4.09 |
4.03 |
4.14 |
4.38 |
| SPCH INDIRECT MEASURE |
3.74 |
4.15 |
4.22 |
4.42 |
| JMC GRAD. INDIRECT |
(4.03) |
----- |
----- |
----- |
| SPEECH GRAD. INDIRECT |
(4.01) |
----- |
----- |
----- |
- demonstrate necessary technological skills specific to a degree in communication.
| |
2006
|
2005 |
2004 |
2003 |
| OVERALL MEAN |
4.20 |
4.00 |
3.95 |
4.17 |
| JMC DIRECT MEASURE |
4.73 |
4.44 |
4.67 |
4.60 |
| JMC PORTFOLIOS DIRECT MEASURE |
4.33 |
4.13 |
4.45 |
4.19 |
| SPCH DIRECT MEASURE |
4.27 |
3.81 |
4.11 |
3.80 |
| JMC INDIRECT MEASURE |
4.18 |
3.96 |
3.89 |
4.40 |
| SPCH INDIRECT MEASURE |
3.49 |
3.66 |
3.31 |
3.62 |
| JMC GRAD. INDIRECT |
(3.55) |
----- |
----- |
----- |
| SPEECH GRAD. INDIRECT |
(2.91) |
----- |
----- |
----- |
SECTION II: PARTICIPANTS, EVALUATORS AND PROCEDURES
Capstone Course – this is tentatively scheduled to be offered for the first time during the spring semester of 2008. Some of the evaluation methods employed in various department venues (Portfolio Reviews, Speech Presentations, Student Perceptions) will be conducted in the capstone when it is offered.
Speech Presentations – evaluations of presentations (N = 11) made by seniors enrolled in 400 level Speech classes during the 2005-2006 academic year were conducted using the direct measure of undergraduate competencies. Communication faculty assessed the speech presentations and care was taken to ensure that a faculty member would not evaluate his or her own class.
Student Perceptions – students in 400 level Speech (N = 17) and JMC (N = 39) classes during the fall and spring semesters were administered the indirect measure of undergraduate perceptions.
Portfolio Reviews – mass communication students graduating May and December 2006 submitted portfolios during the fall semester of 2005 and the spring of 2006. Their portfolios were analyzed as part of the unit’s assessment activities. Communication faculty participated in the evaluation of the senior portfolios. A total of 96 evaluations of approximately 48 portfolios were completed using the direct measure of undergraduate competencies.
Multimedia Presentations – senior-level Multimedia majors and minors (n = 6) on two teams were enrolled in JMC 498 during the spring semester of 2006. There were 11 evaluations of two capstone projects using the direct measure of undergraduate competencies. Several Communication faculty members and outside observers evaluated these Multimedia presentations. Care was taken to ensure that the instructor would not evaluate her own class.
Student Teaching – during 2005 - 2006, there were not any Communication majors who were student teaching.
Communication Graduates – the survey of graduates was conducted summer 2006. A total of 91 Communication graduates responded to the on-line survey (JMC N = 53, SPCH N = 38).
SECTION III: FINDINGS
GRADUATE PERCEPTIONS
Mass Communication and Speech graduates believed that the department was successful at broadening their intellectual interests, helping them understand that it is the right of citizens to voice opinions even if they are unpopular ones, teaching them to work with others to complete tasks and getting them to understand the importance of meeting deadlines. They neither agreed nor disagreed that their experiences were successful in 11 of the 20 areas explored (see Appendix A: Indirect Measures of Graduate Perceptions). This could be a function of (1) forgetting what they learned while here, (2) department goals and objectives formulated since they graduated and don’t apply to what and how they were taught or (3) few strong beliefs in the success of the unit’s efforts. Graduates will be surveyed again in 2011.
UNDERGRADUATE PERCEPTIONS
Overall, current undergraduate students agreed that the department’s programs have helped them in positive ways in areas such as becoming more comfortable speaking to groups, gaining knowledge of communication theories and understanding that it is the right of citizens to voice opinions even if they are unpopular (see Appendix A: Indirect Measures of Undergraduate Perceptions). On the other hand, students neither agreed nor disagreed that the department’s programs were successful at helping them learn: the history of the field; leadership skills and; how to make good decisions.
In some parts of the survey, responses were mixed and areas of strength were related to the respondents’ academic program. Speech majors, more so than Mass Communication majors, felt that the department was successful at helping them analyze oral arguments. Mass Communication majors, more than their counterparts, believed that the unit was successful at helping them: develop strategies or solutions to problems in their field; learn to write clearly; analyze written, oral and mediated messages; broaden their intellectual interests and learn about current issues in the field; work alone to complete tasks and meet deadlines; acquire technological skills and learn about design principles and; engage in information gathering.
In 2005 – 2006, it would appear that, in terms of the department objectives and the means derived from only the indirect measures for each of the goals, Mass Communication and Speech undergraduates agreed that the department was successful at meeting Goal 4 but there was not agreement among them regarding its success at achieving Goals 1 and 5. Mass Communication and Speech students were united in being undecided regarding the department’s success at achieving Goals 2, 3 and 6.
FACULTY-EVALUATOR PERCEPTIONS
Students in both areas of the unit improved over last year, at least in the eyes of evaluators. The data from the Mass Communication direct measures suggest that the evaluators believe that the department was successful at achieving all measured goals -- 1, 2, 3, 5; Goal 4 was not evaluated using the direct method. The data from the Speech direct measure, indicate that the evaluators believed that the unit was successful at achieving Goals 1, 5 and 6 but were undecided about 2 and 3. Goal 4 was not measured using the direct method (see Section I: Department Goals. See also, three different direct measures instruments in Appendix A).
In terms of the Speech and Multimedia presentations, evaluators agreed that students were successful at: logically organizing a clearly understood presentation; using proper spelling and grammar; developing well-researched presentations; working independently and; demonstrating that they developed the technological skills needed to be successful.
Regarding the senior portfolios, evaluators rated all aspects of the materials positively. The appointment of a Portfolio Coordinator has eliminated many problems faced in the past.
SECTION IV: ASSESSMENT OF THE ASSESSMENT PLAN
In terms of the department’s success at achieving its goals during the 2005 – 2006 academic year, five means reached the 4.00 threshold and, as a result, the unit met Goals 1, 2, 4, 5 and 6. Further, it nearly reached that standard on Goal 3. Obviously, there is always room for improvement. The overall means for two of the six objectives were lower than those of 2004 – 2005.
The greatest difficulties faced by the department’s assessment committee concern the presentation of the results to the faculty in a timely manner which, preferably would be prior to the start of the fall semester. This would allow instructors the opportunity to modify syllabi and course materials at the beginning of the school year. Last year, the presentation did not occur until February and this may have been due to a miscommunication. The second problem concerns educating Communication faculty about the meaningful relationships between their efforts as instructors, their course content and the data. One tactic that will be employed this year is to reduce the size of the assessment report for the faculty down to two or three pages to make it easier to digest. The full report will be available to anyone wishing to see the complete data set. The final major problem is making the assessment results top-of-the-mind for instructors. Discussions about how to do these things will continue in the committee and in department meetings.
SECTION V: RECOMMENDATIONS FOR FUTURE IMPROVEMENT
Changes to classes and sequences are not warranted at this time. The adjustments that need to take place concern the efforts of teachers and the modification of instructional materials and classroom activities. A meeting to discuss results of the 2005 – 2006 assessment program with faculty has been scheduled for late-September at which time a shortened version of this report will be distributed. Ideas relating to application of the data to everyday instructional activities will be discussed.
SECTION VI: ASSESSMENT OF THE ASSESSMENT PROCESS
The direct and indirect instruments should not be modified for a period of time (i.e., two years) to eliminate the possibility that future fluctuations in the means are the result of instrumentation and not academic adjustments.