University of Nebraska Kearney

English Assessment Reports 2006

Submitted Fall, 2006

Writing Samples Assessment

This was the seventh year of our implementation of the new system of assessment (uniform rubric, protocols, etc.) and the fourth using a descriptive rubric in this assessment.  As in past years, we performed further analysis by looking at the pattern of results in the four writing areas being assessed: content, organization, mechanics/usage, and language.  We used a 16-point scale to grade the writing samples, with a range of 1-4 in each of the four component categories.  The results are recorded in Table I, below:

We now have nine years of data, seven of them using the current data collection methods.  The A-level score of 9.120 in 2005-2006 reversed a steady decline in A-level scores from 1996-2004.  The B-level score of 10.19, however, represents a sharp decline from 12.08 in 2003-2004.  Because the cohort group of 2003-2004/2005-2006 started with a significantly lower score than that of 2001-2002/2003-2004 (7.44 points as compared with 9.51), it ended up with a lower score as well.  See Tables II and III, below.

Table II

Table III

The English majors and minors assessed during 2005-2006 showed a marked improvement in the quality of their writing between their second and fourth years.  The average score for ENG 234 (A-level) papers in academic year 2003-2004 was 7.44.  The average score for Senior Seminar (B-level) papers in academic year 2005-2006 was 10.19, yielding an average improvement of 2.75 points in the cohort group.  This level of progress compares favorably with that of every other year for which we have assessment numbers save one (the cohort group of 1996-1997/1998-1999).  The 2003-2004/2005-2006 cohort group not only raised its overall score, it raised all four of its component scores as well.  See Tables IV and V, below.

Cohort Group for 2003-2004/2005-2006:

Table IV

Historical Comparison of Cohort Groups, 1996-2006:

Table V
Comparison of Cohort Groups, between Sophomore (A) and Senior (B) years

Perhaps unsurprisingly, individual variation presents the greatest challenge for analysis.  Many students showed dramatic improvement.  For instance, while only one in twenty students (5%) attained a total score of “11” or higher on their A-level papers, thirteen of twenty-eight students (46%) attained such a score on their B-level papers.  Nevertheless, some students are clearly being left behind.  Whereas five of  twenty students (25%) received a “mechanics/usage” score below “2” on their A-level papers, seven of twenty-eight students (25%) received a mechanics/usage score below “2” on their B-level papers—the percentage of those scoring under two remained the same.  Indeed, mechanics and usage showed the least average improvement of all four categories, 0.406 points (see Table IV, above).  As instructors, we may assume that all upper-level students recognize the importance of mechanics in formal writing, but such an assumption is not, unfortunately, borne out by the numbers.  On the other hand, content scores improved significantly.  While thirteen of twenty students (65%) earned a content score below “2” on their A-level papers, only eight of twenty-eight students (28%) earned such a score on their B-level papers.  As in previous years, students showed the most improvement in organization (Table IV, above). 

Assessment of the Assessment

As we noted last year, the “validity of the cohort model of comparison . . . depend[s] on uniform progress through the major, which has not always been the case.  Although the number of students taking ENG 234 as juniors continues to diminish, a few transfer students may still be taking ENG 234 as upper classmen.  Furthermore, students who take a 400-level seminar before they are seniors because it might not be offered again before they graduate may contribute to lower scores at that level.  Thus we are still likely to be assessing some Level 4 papers written by juniors rather than graduating seniors.”  In addition, although ENG 234 is designed for English majors, some non-English majors take ENG 234 to fulfill the General Studies requirement, so we may be assessing a handful of non-English majors at the A-level.  The department has discussed withdrawing ENG 234 from the General Studies pool, which would rectify this problem.

On a different note, the method of requiring “contiguous scores” has become more of a hindrance than a help.  With such a wide range of possible scores (4-16), it is impractical to expect two readers to have touching scores (5-6, 10-11, etc.).  In 2005-2006, a total of seven A-level papers needed to be sent to a third reader because the first two readers’ scores were separated by only two points (for example, 6-8, 10-12).  The total scores for these seven papers ended up being 67.  If we had simply averaged the first two readers’ scores (6-8=7, 10-12=11), the total score for these papers would have been 72—less than one point higher per paper.  As for the B-level essays in 2005-2006, two papers needed to be sent to a third reader because the first two readers’ scores were separated by two points.  The total scores for these papers ended up being 16.5; averaging the first two scores for each paper would have yielded a total score of 19 points, a difference of 1.125 points per paper.  Such differences, especially when they are distributed over the entire cohort, are negligible.  The assessment committee should, therefore, consider adopting a policy of averaging  the two scores rather than sending the paper out to a third reader when the two readers’ scores are separated by only two points. 

A.  INDIVIDUAL PORTFOLIO ASSESSMENT

Individual portfolio assessment was not undertaken this year.

B. STUDENT TEACHING EVALUATION ASSESSMENT PLAN

Student teaching evaluation was not undertaken this year.

C. STUDENT TEACHING SELF-EVALUATION ASSESSMENT PLAN

No surveys were distributed during the 2005-2006 academic year.

D. CAPSTONE COURSE PERSONAL ASSESSMENT PLAN

No surveys were distributed during the 2005-2006 academic year.

E.  ASSESSMENT OF THE ASSESSMENT PLAN

Assessment of English majors and minors was limited to direct measures—that is, the writing samples—this year because the English department was also heavily invested in undertaking its first assessment of General Studies writing and literature courses, which necessarily consumed much of the department’s assessment committee’s time.  Given the fact that not all measures need to be implemented each year, portfolio comparisons and indirect measures will be resumed in future years on a rotating basis.

1. INDIVIDUAL PORTFOLIOS
   Scored essays are being retained for comparison of individual student progress.  To insure sufficient numbers of matched essays, the committee should consider completing portfolio assessment in four-year intervals.
    
2. STUDENT TEACHING EVALUATION
   To insure annual collection and accurate analysis of data provided by COE, a sub-committee of the English department assessment committee should be formed to work with COE faculty.
    
3. STUDENT TEACHING SELF-EVALUATION ASSESSMENT PLAN
   Since collection of data from the Student Teaching Self Evaluations has proved difficult in the past, the assessment committee should seek help from Information Technology Services staff to develop an online survey. 
    
4. CAPSTONE COURSE PERSONAL ASSESSMENT
   Development of an online survey should be pursued for the capstone course personal assessment as for the Student Teaching Self Evaluations.
    
5. MODIFICATIONS OF THE ANNUAL PLAN
   Based on discussions with administrators in the Office of Assessment who pointed out that assessment of the various English department programs—General Studies, majors and minors, and graduate students—may be alternated on a rotating basis, the department should consider devising a workable schedule for assessing each of its respective student groups.  In cooperation with the Composition Committee and the Graduate Committee, a regular cycle of assessing one program per year should be devised to insure the most effective use of faculty time and department resources.