University of Nebraska Kearney

English Graduate Program Assessment Report 2006

Submitted Fall, 2006

Department of English
Graduate Program Assessment Report
October 2006

Assessment Mission Statement:

Students completing the Master's Degree in English--with areas of specialization in literature and literary analysis or in creative writing--will have a deepened knowledge and appreciation of world, British, and American literatures (including multicultural literatures); they will have achieved a greater mastery of the language and of critical thinking, through study of linguistics, grammar, literary criticism, and composition/rhetoric; they will have honed their creative abilities in the areas of poetry, fiction, drama, and creative nonfiction; and will have acquired habits of speaking, listening, reading and writing that are the essence of reflection, dialogue, and academic discourse.

Goals: English MA recipients will be able to conceive, undertake, research, and complete extended projects (whether through the thesis option or the portfolio option) that indicate a much greater mastery of arts and skills than is expected at the undergraduate level. Through these endeavors, students will enhance their personal lives, their knowledge of their world, their ability to make a difference, and their personal sense of values.

In addition to the students' life-long learning, they will reach a level of competence sufficient to be able to take the next step, whether that be into the professions, into more secure and useful roles as teachers, into jobs that value creative and critical thinking and communications, or into further study in doctoral or MFA programs.

Objectives: A student who completes the MA program will demonstrate (through discussion, class assignments, and extended projects and, for some, through their own teaching, tutoring, or assistance with editorial and research projects):

Knowledge and Skills:

1. Proficient levels of knowledge of literary periods, creative genres, and the language of literary and creative study.
2. The mastery of skills and arts necessary to creative, expository, or critical writing.
3. The ability to undertake and present textual analysis and discourse analysis from a variety of theoretical positions.
4. The capacity to think and communicate clearly and objectively.
5. The ability to employ research methods and critical terminology.

Values:

1. A readiness to connect literature and creative writing to humanity's greatest concerns.
2. Some awareness of how persons think, learn, create, and evaluate.
3. A life-long commitment to their own literacy and that of others, including the experience of literature's and language's joys and constraints.
4. An intellectual curiosity and honesty and an appreciation for the private and public roles of the intellectual life and of intellectual communities.

Direct and Indirect Assessment Measures:

The English Department uses the following measures for Graduate Program Assessment:

Direct Assessment Measures:

Graduate Student Portfolios
Writing Samples

Indirect Assessment Measures:

Graduate Student Personal Assessment Surveys
Department Exit Surveys

Direct Measure #1: Graduate Assistant Portfolios
(Knowledge/ Skills Objectives 1, 2, 3, 4, 5; Values Objective a, b, c, d)

Each year, current graduate assistants are required to submit a portfolio. This portfolio (and its assessment) is required for requests to the Graduate Office for the continuation of the assistantship, the renewal of the position, and/ or the creation of a new assistantship position. This portfolio includes the following items:

Current curriculum vitae
Self-assessment statement
Statement of teaching strategy, goals, philosophy, and accomplishments
Written description of class visitation by the Composition Coordinator
Statistical scanning sheets for student evaluation of courses (and, as an option, any written  
     student remarks)
Evidence of participation in scholarly, creative, and/or service activities (such as conferences,
     presentations/readings, publications, community service)

Students may also submit comments pertinent to their academic performance.

The Composition Coordinator and Graduate Director (and any other supervisors or mentors to whom the assistant may be assigned) submit letters of evaluation. In addition to verifying that the assistant has maintained a GPA of 3.0 or higher and demonstrated sufficient progress to retain the assistantship position, these evaluations examine the portfolio materials for general evidence of the knowledge and skills [see 1-5 under “Assessment Mission Statement”] and of the values [see a-d under “Assessment Mission Statement”] included in the program objectives. They are particularly useful in assessing the values objectives. As the evaluations touch on the elements of the portfolio, they comment specifically on the following elements of the portfolio:

1) The extent to which the current curriculum vita shows the ability to communicate clearly and objectively and presents the unique student in an accurate and persuasive manner;

2) The extent to which the self-assessment statement portrays a student with a commitment to life-long learning and an ability to connect academic and ethical concerns, and presents a unique and curious individual of integrity seeking to play a role in intellectual and community forums

3) The extent to which the statement of teaching strategy, goals, philosophy, and accomplishments incorporates the student's literary knowledge, critical ability, and acquaintance with appropriate rhetorical, research, and critical practice and terminology; shows creative assignments and an engaging and nurturing pedagogy; embodies accurate understanding of the courses taught and their place in the curriculum; and is borne out in the written description of class visitation by the Composition Coordinator

4) The extent to which the statistical scanning sheets for student evaluation of courses (and, as an option, any written student remarks) and the statement of teaching strategy, goals, philosophy, and accomplishments indicate that the teacher's and the students' perceptions of the classroom experience are similar;

5) The extent to which evidence of participation in scholarly, creative, and/or service activities (such as conferences, presentations/readings, publications, community service) portray an active engagement in the profession and verify a willingness to contribute to the profession, the literature of the field, and the greater public.

For Assessment purpose, each portfolio is reviewed by one of the members of the department Graduate Program Committee, and scored according to the Graduate Assistant Portfolio Assessment Rubric (see Appendix A). In each category, readers score the portfolio on the following scale:

1 (low)

2

3

4

5 (high)

Direct Measure #2: Graduate Student Writing Samples
(Knowledge / Skills Objective 1, 2, 3, 4, 5)

The department Graduate Handbook states that this assessment is conducted each semester, with the results filed for longitudinal studies and conveyed to appropriate committees (e.g. the Graduate Committee) for program improvement. Given the other levels of assessment that also demanded labor-intensive evaluation of student writing, papers from the last academic year are at this point not scored; the data used in this report comes from the 2003-2004 and 2004-2005 academic years.
The guidelines for evaluating sample essays are as follows:

1. Each essay is identified by a number system only (keyed to student name held in (separate file)
2. The committee begins, before assessing essays, reaching agreed understanding of the protocol (norm referencing) for each evaluated category [see protocol below]
3. Two members of the committee--or, in cases where the scores of the two readers are not consistent (not touching scores), three or four readers--read each essay and score it independently
4. Final scores are calculated as an average of all scores received.

Indirect Measure #1: Graduate Student Personal Assessment Surveys
(Knowledge/ Skills Objective 1, 2, 3, 4, 5; Values Objectives c, d)

Over the course of a graduate student's program in English, the Assessment Committee administers a Graduate Student Personal Assessment Survey similar to that given at the undergraduate level in the capstone courses. The department Graduate Program Committee uses these self-assessments to determine the level of graduate student satisfaction with the overall learning experience and the component parts of the graduate program in English.

The survey asks the student 1) to rate his/her proficiency in a number of areas: Literary periods; Literary genres; Literary terms and techniques; Composition and essay writing; Creative writing; Language and grammar; Communication skills; Critical thinking; Literary appreciation; Teaching methods and materials; Research methods; Critical approaches and 2) to rate to what extent have our English classes contributed to: Subject matter knowledge of literature, its backgrounds, history, and values; Intellectual curiosity; Desire to explore and help others experience language's pleasures and constraints; Competency in research methodologies; and Proficiency in electronic and technology literacies. The survey, included below, also asks the student to provide open-ended feedback about the strengths and weaknesses of the program.

Indirect Measure #2: Graduate Student Exit Interview
(Values Objectives a, b, c, d)

After completion of the thesis defense or comprehensive exam, students are asked to complete an exit survey of open-ended questions soliciting information about and feedback on various aspects of their experience in the program. The survey asks: 1) how the student learned about the program and why s/he chose it; 2) what the strengths and weaknesses of the curriculum were; 3) what courses or subjects not offered might have been beneficial; 4)whether the student chose the thesis or portfolio option, and why; 5) what value was derived from experience as a graduate assistant (if relevant); 6) how the educational experiences beyond the curriculum (e.g. teaching/ advising, other students, publications/ presentations) affected the student; 7) what the student’s short-term and long-range plans are; 8) what other additional comments the student has.

Data Report:

During the 2003-2004 and 2004-2005 academic years, assessment was conducted on six (6) graduate assistant portfolios and twenty (20) graduate student writing samples.  Five Graduate Student Personal Assessment Surveys were returned in the same period; an additional four were collected in the 2005-2006 year, giving us a total of nine (9) personal assessment surveys.  Only two of the students who returned the Assessment Surveys from the 2003-2005 groups also completed the department Exit Survey; all four from the 2005-2006 group did so, providing a total of six (6) open-ended surveys from which to report responses.

The English department typically has five (5) Graduate Assistants funded by the Office of Graduate Studies & Research.  Not all of these GA’s teach expository writing, and the ones that do not are not required to produce a portfolio, since some of the items assessed relate to teaching.  On occasion, a particular faculty member will receive funding for a Research Assistant from the Graduate Office; these students do not produce a portfolio.

Total enrollment in graduate courses in Fall 2003 was 39, with twenty (20) different students taking classes.  Of these, ten (10) would be classed as full-time students; the other ten (10) are part-time degree-seeking students and non-degree students. In the Fall of 2004, total enrollment in graduate courses in was 41, with twenty-three (23) different students taking classes.  Of these, ten (10) would be classed as full-time students; the other thirteen (13) are part-time degree-seeking students and non-degree students. 

The department awarded six (6) M.A. degrees in the 2003-2004 academic year (including summer) and nine (9) M.A. degrees in the 2004-1005 academic year (including summer).

Results and Interpretation of Assessment Measures:

Direct Measure #1: Graduate Assistant Portfolios
(Knowledge/ Skills Objectives 1, 2, 3, 4, 5; Values Objective a, b, c, d)

This was our second year to use a rubric and standardized procedures for the assessment of graduate assistants' portfolios. Six (6) portfolios were assessed, which provided us with more data than the 2002-2003 assessment.  Since the English department has five total assistantships per year, and on occasion some of those students assume a position mid-year, we will only have a maximum of five portfolios per academic year to evaluate.

The scores from the 2003-2005 portfolios were much stronger than those from the previous assessment, ranging from 3.0/5 to 4.75/5. Their overall average was 3.95, well above the midpoint.  Four out of the six portfolios scored were from students in their first year of the assistantship; no dramatic deviation was noted between first and second year students, although some of the lower scores in individual categories came from portfolios of those in their first year.

As the chart above shows, the students assessed received the highest marks in H (measuring how well the Statement of Teaching Strategy measures understanding of the course’s place in the curriculum).  The scores in I and G, both of which relate to teaching effectiveness (and which were noted as the lowest in the 2002-2003 assessment) not only showed dramatic improvement but also received scores on the high end of the measure (4.25 & 4, respectively).  It may be that since ENG 805, our course in teaching methods which only receives sufficient enrollment to be taught every other year, had some impact on these particular students, who had recently taken the course, or that individual mentoring of graduate assistants has improved.  The weakest scores came in D (measuring how well the student demonstrates ability to connect academic and ethical concerns) and B (assessing the student’s curriculum vitae).  The former remained at the same average as the previous assessment, while the latter rose .83 points.  Nonetheless, given that these scores are the lowest for this more recent batch of portfolios, it suggests that we might make expectations clearer and provide further guidance in these areas.  Overall, the portfolios were much more professionally assembled and did a better job of addressing the criteria in nearly every area.

Direct Measure #2: Graduate Student Writing Samples
(Knowledge / Skills Objective 1, 2, 3, 4, 5)

The English Department Assessment Committee scored a total of twenty (20) graduate student writing samples from 2003-2005: thirteen (13) from the 2003-2004 academic year and seven (7) from the 2004-2005 academic year (*see note below). The scores ranged fro a high of 19/20 to a low of 11/20—close to the same range that the 2002-2003 data from the previous report yielded (see graph below). Only the composite average of the total scores will be compared in this report, rather than noting scores in each sub-category (content, organization, mechanics/ usage, language, research).

Graduate Writing Samples
Average Composite Scores
02/03 through 04/05

One might group the composite scores into four groups for further comparison with the data from the 2002-2003 Assessment Report:

• three (3) essays in the 11-12.5 range;
• five (5) essays in the 13-14.33 range;
• eight (8) essays in the 15-17 range
• four (4) essays in the 17.5-19 range

Given the rubric that we are using, scores of 1 & 2 in the five individual categories scored would be below the midpoint, while scores of 3 & 4 would indicate performance above the midpoint. A composite score of 12.5 (a 2.5 score for each of the five categories), would thus represent an average performance, while an essay receiving a score of 3 in each category—or a composite score of 15—would be clearly above the midpoint and thus represent an above average performance. It is heartening to note that our average score for the twenty essays scored is 15.19, which brings us above the threshold of a composite score that ranks above average.

The fact that 12/20 essays received a score of 15 or above indicates that our students are learning (or perhaps have learned) to achieve proficiency in the language of literary study, to master the skills necessary for critical writing, to produce textual analysis from an appropriate theoretical position, and to employ research methods and critical terminology capably. Only 3/20 essays were scored in the lower range (12.5 or below), as opposed to 3/9 essays from 2002-2003 (a full one-third of the samples). Yet the group of five essays in the 13-14.33 range indicates that we still have work to do in achieving some of our knowledge and skill objectives. It may also simply reflect that the pool of students from which we recruit enters the program with minimal to average skills, or it may reflect that we do less direct writing instruction at the graduate level—where end-of-term seminar papers area the norm—than we do at the undergraduate level. One might also note that creative writing track students are only required to take 6 hours of literature courses along with the literary criticism course; most of their writing instruction occurs in the creative area, which we are not assessing. However, since we do not identify which essays are from students in each track (literature vs. creative writing), one cannot draw valid conclusions about the source of the scores below 15.

Overall, the composite average for the last two years of writing samples has improved since the last assessment, and we have begun to see more students scoring above the midpoint of the rubric scale.

(*NOTE: For the 2004-2005 academic year, fifteen essays were collected, but eight (8) student essays appear to have received only one reading or two readings that did not yield touching scores that could be averaged. It will be recommended that for the next report, these essays receive the necessary readings and that the data for 2004-2005 year undergo reanalysis).

Indirect Measure #1: Graduate Student Personal Assessment Surveys
(Knowledge/ Skills Objective 1, 2, 3, 4, 5; Values Objectives c, d)

Graduate student self-perceptions continue to show a high level of confidence in communication skills and literary appreciation, and much higher confidence in English language/grammar, composition/ essay writing, and creative writing. Composite average scores range from 5.6/7 (critical approaches to literature) to 6.6/7 (literary appreciation).  This range is not only much higher than the average from 2002-2003 but also shows significant improvement in a number of categories.  Students report increased knowledge of literary periods (+.72) and genres (+.85), with average scores in both areas now approaching 6/7; a significant increase in knowledge of literary terms (+.85/ now above 6) was reported. 

The gains in the areas of perceptions about skills in composition & essay writing (+1.3 points, from 5.2 to 6.5) and language (+1.7 points, from 4.5 to 6.2) were the most dramatic.  As noted in the 2002-2003 report, graduate students rarely take English language or grammar courses, so many of their self perceptions probably stem, at least in part, from professors' comments on writing assignments.  The rise in sores on creative writing (+.67, from 5.2 to 5.87) may derive from the growth of our M.A in creative writing, as nearly half of our full-time M.A. students declare the creative writing track and write creative theses.  Scores in the areas of communication (-.35, from 6.3 to 5.95) and critical thinking (-.15, from 6 to 5.85) decreased slightly, but still remain very close to 6/7 and indicate continued satisfaction in those areas.

Graduate English Personal Assessment        
Spring 2006            
Ten Students

Considering these scores as, in part, a measure of student satisfaction with the program, there has been a strong increase in students' sense of success and accomplishment.  We should seek to understand why perceptions about abilities in communication and critical thinking have undergone a slight decrease, while other areas are now rated more highly.  Continued assessment of these areas should enable the department to better measure whether student satisfaction is actually increasing, but the increase in the number of responses to the survey may be giving us a better measure of our students’ self-analyses of their capabilities at the end of our program.  Our challenge may be to maintain these high student ratings of their capabilities in the knowledge and skills areas.

Indirect Measure #2: Graduate Student Exit Interview
(Values Objectives a, b, c, d)

A range and variety of comments emerge from the open-ended exit surveys; at times, one hears echoes of particular comments, while at others the recommendations or perceptions contradict each other or urge the program to move in different directions.  A number of students chose the program because of their undergraduate experiences in English at UNK or because of some contact with a faculty member in another venue; others began taking classes based on positive feedback from our former students. 

In general, the curriculum was thought to be flexible, to provide a firm base in theory, and to distribute requirements adequately.  Some students wished for more classes in their particular area of interest; creative writing emphasis students wished for more creative writing classes, which were praised for helping students find their voices.  Two students felt that grammar/ linguistics courses, though not required, were very helpful, especially in teaching.  One felt we should offer more linguistics & genre study courses; another wanted to see more world literature and more contemporary literature courses.

Much praise for the faculty was in evidence, especially for the intellectual challenge, stimulating discussions, enthusiasm, guidance, and close personal attention they provided.

All students chose the thesis option, as they wanted experience in producing a professional piece of writing, either for further studies or for professional reasons.  Some commented on the need for further guidance on the thesis, perhaps an early class addressing this component specifically.

Those who were graduate assistants characterized the experience as invaluable, one that made them feel like professionals as they took charge of their own classrooms or worked closely with professors in writing/ editing tasks.

Other aspects of the program that drew positive comments were the small classes, individualized attention, the talks and readings by professional writers, and the manner in which professors functioned as mentors.  A mixed verdict on advising was in evidence, with some stating that they had strong advising, and others noting that advising could be better.

One students suggested that we offer literary criticism more often; another felt that the demise of the Ft. Kearny Writer’s Workshop was tragic and that it should be revived. 

We are, according to one student, “one of the best kept secrets in Nebraska” providing a superb quality program and excellent instruction.  No serious weaknesses were noted, yet most respondents seemed to want more courses in their areas, especially the creative writing students.

Recommendations Resulting from Assessment

Since the assessment of Graduate Assistant Portfolios contains a number of “(N/A)” responses on certain items, indicating that not all students included all the material we felt should be there, some further instruction on portfolio preparation may needed.  In addition, we should provide professional samples of curriculum vita in the field and/ or send students to the UNK Career Center for assistance in creating this important professional document.  We might also consider workshops in particular areas that each year’s assessment shows to be the lowest; in this case that area would involve the connection between academic and ethical concerns.

Since some of the writing samples that might have figured in this assessment did not receive a second score, or did not receive touching scores that would have yielded an average, it is recommended that the academic year does not conclude without all essay scoring completed, or that a summer scoring session be held.

We currently assess analytical writing that concerns literature in our assessment, because all students in the program produce samples in that mode.  However, students in the creative writing track are required to take only two literature classes (and the required literary criticism course), while four creative writing classes are required.  There is also the problem that we are currently assessing multiple papers from the same students, written at various points in their graduate career, with no distinction as to whether they are representative of their skills when they enter our program or near the end of our program.  In addition, it is very labor intensive to read and score a number of twenty-page seminar papers, even if the rubric does provide us with significant information about our student’s analytical writing capabilities . It is recommended that the department Graduate Committee discuss the creation of some other mode of assessing the knowledge and skills objectives that will be applicable across both tracks—literature and creative writing—perhaps some rubric for evaluation of the thesis, done by the student’s thesis committee.  If we choose to keep the current mode of writing assessment, we should discuss modification of the process, including better essay collection methods, indication of when in a student’s career the essay was written, and whether we should modify the policy of obtaining touching scores before an essay’s score is calculated.

The Personal Assessment surveys yield important data on students’ perceptions of the proficiencies in crucial areas of knowledge, skills, and values, and the role that their courses have played in facilitating these.  To be more useful, it must be more systematically administered and collected.  Since it seems that many students who received their degrees in the 2003-2005 academic terms either were not administered the survey or did not return it, a mechanism need to be developed to ensure that these surveys are administered and returned.  Also, students who are easily accessible and who did not fill out the survey should be contacted and asked to do so.  The current Graduate Program Director has adopted a practice of handing out the surveys as the M.A. degree is completed (after the comprehensive exam), which has proved to yield higher rates of return; it is recommended that this practice be continued, with follow up emails until students return both the personal assessment survey and the department exit survey.

Assessment of the Assessment Process: