University of Nebraska Kearney

Modern Languages Assessment Report 2006

Submitted Fall, 2006

Assessment Report for the Department of Modern Languages
September 2006

Undergraduate Assessment (including Foreign Language Assessment for the current General Studies Program)

I. Direct Measures

In preparation for its Academic Program Review (April 2006) and as part of its ongoing assessment of its assessment, the Department of Modern Languages made a special effort to revise and to define more clearly some of its assessment rubrics, especially for the Basic Oral Interview (#1, which is part of its General Studies Assessment), for the Cultural/Literary area (#2, which appeared to be too vague) and for the Final Evaluation of Student Teachers (#5, whose form had changed). Also because of the APR, it was careful to finalize the curricular change suggested in the 2005 Assessment Report concerning the raising of the prerequisite for SPAN 304, 305 and 360 from SPAN 201 to 204.

Furthermore, not only was the Department of Modern Languages able to continue in the 2005-2006 academic year with the assessments that it has done in the past in SPAN 201, 304, 305 and 360, but for the first time Basic Oral Interviews (Objective A with Rubric #1) were conducted also in FREN and GERM 201, which means that the General Studies Assessment is now being fully implemented in our three principal languages.
 
We, of course, should recall that the Basic Oral Interview (which began in Spring 2005 as a pilot program for the oral component of the foreign language portion of General Studies Assessment) entailed interviewing all of the students in the two sections of SPAN 201.  Its results were discussed in the 2005 Assessment Report, and the Department of Modern Languages decided not only to continue this oral assessment program, along with the BYU exam, and to expand it to include French and German.

Moreover French is now planning to start to assess the Cultural/Literary area (Objective B with Rubric #2) in its civilization course (305), and we are also ready to compile the student portfolios suggested in Objective E (Rubric #6).

Objective A (also Foreign Language Assessment for General Studies)

The basic communication skills at the intermediate level (which is the one pertinent for students earning 3 hours of General Studies credit, for students that select the traditional (A) option (6 hour at the intermediate level) for the B.A. degree, and for language students that are beginning their majors and minors) are now being assessed in two ways: 1) their ability to understand the written language and its structures is tested through the Brigham Young (computer adaptive) Exam, and 2) their ability to understand the oral language and to respond to it in speech is assessed through the Basic Oral Interview.

1) Students in FREN/GERM/SPAN 200 are expected to take the BYU Exam at the beginning of each semester when these courses are offered, and students in FREN/GERM/SPAN 201 need to take the same (but variable) exam at the end of the latter courses. According to the ML Assessment Plan (see V. Rubrics and Interpretation of Data  #1, “individual students are expected to improve their score by at least 10% and the average improvement should be 15%.”

In Fall 2005 and Spring 2006 six French students took both exams. In FREN 200
the scores ranged from 34 to 447, and the average score was 276.7, and in FREN 201 the range was from 153 to 514, and the average was 341.8. The average improvement was 71%, which is well above the expected 15%. Five of the six students exceeded the 10% individual improvement goal (the highest was by 350%) but two did not, making the success rate 83%.
 
In 2005-2006 four German students took both exams. For GERM 200 the range was from 223 to 521, and the average score was 387.8, while for GERM 201 the range was from 396 to 592, and the average score was 502.3. The average improvement of these students was 41 %, which is also higher than the expected 15%.  Three of the four students improved their scores by at least 10%, thus the success rate was 75%.

Since SPAN 200 and 201 are offered every semester (unlike for French and German), we were able to assess two groups: the fifteen students that took the two-course series in Spring 2005 and Fall 2005 and the twenty students that began in Fall 2005 and ended in Spring 2006 (please note that other students were enrolled in these courses but did not take both exams so they could not be included in this assessment study). In the first case, the range for SPAN 200 was from 172 to 398, and the average score was 299.4, while the range for SPAN 201 was from 230 to 462, and the average score was 351.1. The average improvement was 22.3%, which is above the expected 15%. Of the fifteen students, ten exceeded the 10% individual improvement goal (with the highest being 84.6%), but five did not. Thus the success rate was 66%. For the second group (Fall 2005 and Spring 2006), the range for SPAN 200 was from 151 to 375, and the average score was 292. In SPAN 201 the range was from 273 to 493, and the average score was 362.5. The average improvement was 29.1% (with the highest being 95.9%), which was somewhat higher than for the previous group. Likewise for the 10% individual improvement goal the students also did somewhat better because nine of the twelve students were successful, while three were not, making the average 75%.

As in the report of the previous two years, all three languages met the 15% average improvement goal. We can see these by language: French 2004: 16.6%, 2005: 24.7%, 2006: 71%; German : 2004: 20.5%, 2005: 24.5%, 2006: 41%; Spanish 2004: 31.8%, 2005: 17.5% & 21.1%, 2006: 22.3% & 29.2%. (It should be noted that the small number of students for French and German may imply less consistency, particularly when, as in this year one of the French students started out very low and increased his score by 350%.) As for the 10% individual improvement expectation, we can also note the following results for the past three years: French 2004: 60%, 2005: 67%, 2006: 83%; German 2004: 75%, 2005: 100%, 2006: 75%; Spanish 2004: 72.7%, 2005: 60% & 65%, 2006: 66% and 75%. We can conclude that French and Spanish are higher than last year. Although German is not as good as in 2005, it is the most consistently high. We should note that one of the problems, particularly for Spanish (but not exclusively so), is that, because of carelessness or some other reason, from two or four students in the past two years have received a lesser score in 201 than in 200, which lowers both the percentage of average improvement and the percentage of students reaching the individual expectation level. In the future such students should be encouraged to perform as well as they can on both exams. Otherwise the results are not accurate, which serves neither them, the Department of  Modern Languages or the General Studies Program.

2) Students completing FREN/GERM/SPAN 201 demonstrate their basic speaking proficiency level in the previously described Basic Oral Interview, which is conducted by their instructor or another faculty member of the appropriate language. Also according to the revised Assessment Plan (again see V. #1, and note that the general levels are the following: 1: Novice, 2: Intermediate, 3: Advanced, etc.) the students are expected to perform at the level of Intermediate-Low (1.8), Mid (2.0), or High (2.5).

In both Fall 2005 and Spring 2006 the students in the two sections of SPAN 201 were interviewed by their instructors. And the students in FREN 201 and GERM 201 were interviewed by their instructors in Spring 2006 (please note that these courses are only offered in the spring). Furthermore as suggested above, before the interviews in Spring 2006 the rubric was more clearly defined, and the interviewers were informed of its details.

As in Spring 2005, Dr. Hart and Professor Arrieta interviewed in Fall 2005 their thirty-one combined SPAN 201 students. The scores ranged from 1.0 (Novice) to 3.5 (Advanced-High), and the average was 2.15. Thirty students reached the expected level of 1.8 (Intermediate-Low), but one did not, making the success rate 98.8%. In Spring 2006 Dr. Pearson and Dr. González interviewed  a total of twenty-six SPAN 201 students. The range was from 1.5 (Novice-High) to 3.0 (Advanced), and the average was 2.8. Twenty-four reached the expected minimum of 1.8 but two did not. Thus the success rate was 92.3%.

In Spring 2006 Dr. Kropp interviewed nine FREN 201 students. The low score was 1 (Novice) and the high score was 2.5 (Intermediate-High). Their average was 1.9. Seven of the students attained the expected level of 1.8, but two did not. Thus their success rate was 77.8%. Dr. Snider conducted the interview for GERM 201 with four students. Their scores ranged from 1.8 (Intermediate-Low) to 2.8 (Advanced-Low), and their average was 2.4. All the students reached the expected level of 1.8 so their success rate was 100%.    

Observing the averages by language (French: 1.9, German: 2.4, Spanish: 2.15 and 2.8) and the success rates (French 77.8%, German: 100%, Spanish: 98.8% and 92.3%), we can conclude that our beginning majors and minors are meeting our expectations for the Basic Oral Interview and are prepared to proceed to our advanced level courses. Likewise the intermediate language students for General Studies credit and for the traditional option of the B.A. degree are able to participate actively in a basic conversation in the language that they have chosen.

Objective B

In order to demonstrate the understanding and appreciation that they have acquired concerning French & Francophone/Germanic/Hispanic cultures, the students in FREN/GERM/SPAN 305 (French/German/Spanish Civilization) or SPAN 360 (Latin American Civilization)--one of  which is required for all language majors and minors-- write a reflective essay on the cultural contributions of the appropriate area.  According to the Cultural/Literary Rubric (see Assessment Plan V # 2; also note that the general levels are the following: l: Novice, 2: Intermediate, 3: Advanced, 4: Superior, 5: Native), “the students are expected to perform at the Intermediate-High (2.5), Advanced-Low (2.8) or Advanced (3.0) level.”

In Fall 2005 Dr. Pearson had his thirty-five SPAN 360 students and in Spring 2006 Dr. Hart had her twenty SPAN 305 students write reflective essays to show their knowledge of the contributions of Latin American and Spanish cultures respectively. It should be noted that in the fall Dr. Pearson used the former version of the Cultural/ Literary Rubric, while in the spring Dr. Hart followed the new expanded version of it. In the previous case the scores ranged from Novice-high (1.5) to Superior (4.0) and the students averaged 3.2 on the rubric scale. Thirty of the thirty-five students reached the minimum expected level of Intermediate High (2.5) but five did not. This means a success rate of 85.7%. In the case of SPAN 305, the range was from Intermediate-Low (1.8) to Superior (4.0), and the students averaged 2.7. Nineteen of the twenty students met with the minimal expectation (2.5) but one did not. Here the success rate was even higher: 95%.

Comparing these two groups with the statistics for the previous two years (Fall 2003 and Fall 2004), we can observe considerable consistency. The lows and highs for all four are very similar (03F: 1.5 & 3.5, 04F: 1.8 & 4.5, 05F: 1.5 & 4.0, 06S: 1.8 & 4.0). Although the averages vary to some extent (03F: 2.7, 04F: 2.6, 05F: 3.2, 06S: 2.7) as do the rates of success (03F: 90.5%, 04F: 71.4%, 05F: 85.7%, 06S: 95%), they all appear to be satisfactory. Since the new prerequisite for SPAN 305 and 360 (i.e. SPAN 204 instead of 201) is only being required now in Fall 2006, we cannot yet gauge its effect upon the students either in terms of average or success rate. In contrast, Dr. Hart indicated that for her the revised rubric was easier to apply.

Objective C

To demonstrate their speaking and writing proficiencies in Spanish, students in SPAN 304 Advanced Conversation and Composition (which is required for all Spanish majors and minors) take both a speaking and a writing proficiency exam (see Assessment Plan V. # 3, please note again that the general levels are the following: 1: Novice, 2: Intermediate, 3: Advanced, 4: Superior, 5: Native). For speaking, “the students are expected to perform at the Intermediate High (2.5), Advanced (3.0) or Superior (4.0) level,” while for writing, “the students are expected to perform at the Advanced-Low (2.8), Advanced (3.0), Advanced-High (3.5) or Superior (4.0) level”.

As has been our normal procedure since 1999, for the sake of reliability the professors of the two sections of SPAN 304, Dr. Lon Pearson and Dr. Herbert Craig, gave in Spring 2006 an Oral Proficiency Interview to each other’s students. All together thirty-five students were interviewed. The scores ranged from 2.0 (Intermediate) to 5.0 (Native), and the average was 3.3. Thirty-two students reached the expected minimum of 2.5 (Intermediate-high), but three did not. Thus the success rate was 91.4%. Since we began to consider last year raising the expected level to 2.8 (Advanced-low), we ought to look at what the success rate would be with this change. Twenty-four students did reach this minimum but eleven did not, which would lower the success rate to 68.6%.

Looking at the statistics over the past six years and considering the success rate at the current and the possibly increased level, we can see much consistency but also some improvement:

Given that NCATE now expects beginning language teachers to have an oral proficiency of Advanced-Low (2.8) and the Nebraska Department of Education is about to finalize its acceptance of an identical standard for its starting language teachers, the Department of Modern Languages decided that its expected level should also be raised from 2.5 to 2.8 so that it may also reflect the new higher standard. Since we have already increased the prerequisite for SPAN 304 from 201 to 204, we believe that our students should be able to meet this new challenge.

For the second part of Objective C, both Dr. Pearson and Dr. Craig gave their SPAN 304 students a Writing Proficiency exam, and following the rubric, each assessed his own students. The range for the thirty-five total students was from 2.0 (Intermediate) to 4.00 (Superior) and their average was 3.1. Twenty-nine students reached the expected minimum level of 2.8 (Advanced-Low) but six did not, which made their success rate 82%. We can observe that the current average of 3.1 ties with the best of the years that we have been assessing writing (2001: 2.71, 2003: 2.4, 2004: 3.1, 2005: 2.85). Also using 2.8 as our point of comparison (even though prior to 2005 the level of 3.0 was our goal) we can see that our current success rate of 82% is the highest yet (2001: 44.4%, 2003: 36.4%, 2004: 71.4%, 2005: 69.2%). We have not seen the results for writing of our recent change of the prerequisite for SPAN 304 from 201 to 204, but this should further improve both the average score and the success rate. 

Objective D

According to the current Modern Languages Assessment Plan, the final UNK Student Teacher Evaluation Form as filled out by the cooperating teacher(s) for each student teacher serves as the basis for this assessment. For the subcategories of the three general areas (Lesson Planning, Instructional Delivery and Teaching Dispositions) the cooperating teacher is expected to evaluate 45 items on a four point scale. The levels include Beginning (1), Progressing (2), Proficient (3) and Advanced (4). The maximum possible score is 180 (45 x 4). Like the College of Education, Modern Languages will accept the individual minimum satisfactory score of 117 (2.6) but will expect for the purposes of assessment an average annual score of 135 (3.0) for all of its graduating student teachers.

In Fall 2005 there were three students that did their student teaching for a foreign language endorsement: one in French 7-12, one in Spanish 7-12 and one in Spanish Middle Grades. The two in Spanish received from the cooperating teacher from good to excellent ratings (139 and 165 respectively), however the other one for unclear reasons received a much lower evaluation. Since the university supervisor also made a separate evaluation of this student, we have decided to blend the two scores into one: 119. In Spring 2005 two of our students performed the practice teaching for their Spanish 7-12 endorsements (plus three others in the post baccalaureate program, which we will not consider here because they did not take their language courses with us). Our two students received ratings of 137 and 156. Thus 100% of the five students for the 2005-2006 academic year surpassed the minimum satisfactory score of 117, and their average of 143.2 was also higher than the expected average score of 135. Although because of the change in evaluation form and the manner in which we are now viewing the current one is different from last year, we can also observe that, with the exception of the one weak student, all of the individual items on the rating sheets were considered to be at the level of proficient (3) or higher. In short, it appears that our endorsement students are well prepared to become successful foreign language teachers. 

II. Indirect Measures

The survey forms for students completing an undergraduate program in Modern Languages at UNK (see the Survey at the end of the ML Assessment Plan) were sent to graduating major, endorsement and minor students immediately after the Fall 2005 and the Spring 2006 Commencements. In the former case this meant contacting six majors, two endorsement graduates and twelve minors, while in the latter case there were three majors, two former endorsement students and ten minors. Fortunately the return rate was much better this year than last. In the fall the two endorsement graduates and three of the minors sent us their completed forms, and in the spring one major, two endorsement graduates and three minors returned their forms.

In their self-assessment the two endorsement graduates of the fall considered their speaking and writing proficiency advanced and their cultural fluency superior. As for their knowledge of literature and of foreign language teaching methods, one rated these as advanced and the other superior. Two of the minors judged that their level was intermediate in all four categories, but the other one considered his speaking and writing to be at the advanced level, his cultural fluency to be superior and only his knowledge of literature to be intermediate.

Among the graduates that completed the self-assessment in the spring, we can note the following: The one major saw his speaking proficiency to be superior, his writing proficiency and cultural fluency to be advanced and his literary knowledge only intermediate. Having studied translation-interpretation, he also marked this area, which he saw to lie between advanced and superior. The two endorsement graduates considered their speaking proficiency and cultural fluency to be superior, but their writing proficiency and knowledge of literature to be advanced. In the case of one of them, she found her knowledge of foreign language teaching methods and of translation-interpreting skills to be superior, but the other simply advanced. Two of the minors rated their speaking and writing proficiencies, as well as cultural fluency and literary knowledge to be advanced, while the third saw her two language proficiencies to be only intermediate, her cultural fluency advanced and her literary knowledge superior.

In the comments written at the end by the students from both semesters, it is evident that all of them are quite satisfied with their modern language classes, activities and professors. Although they offered some suggestions (such as more advanced language instead of literature classes, more study abroad options, a pairing of native and non-native students), all seem to believe that their language study will help them in their future professions.