Instructional Factors that Can Contribute to Course Engagement
Richard Miller
UNK Department of Psychology
 Student engagement is an important predictor of student achievement. While student engagement can take many forms, the aspect of engagement that faculty have the most influence on is course engagement. Course engagement is a multifaceted construct and includes four distinct types of engagement: skills (e.g. making sure to study on a regular basis), participation/interaction (e.g. asking questions when I don’t understand something), emotional (e.g. finding ways to make course material interesting to me), and performance engagement (e.g. getting a good grade). Handelsman and his colleagues (2005) developed a student course engagement questionnaire (SCEQ) to measure each of those dimensions.
The Psychology Department has recently completed a study that examined instructional factors that can contribute to course engagement. Of particular interest was the impact of faculty-determined factors (e.g. class size, teaching style, responsiveness to questions) on student engagement. Also of interest was the role of student motivation (e.g. intrinsic vs. extrinsic motivation) in mediating engagement and whether interactions exist between faculty-determined factors and student motivation factors. Finally, differences between faculty members in the type of engagement each was more likely to promote was of interest. The SCEQ was administered to all students enrolled in psychology courses at UNK at the end of the fall semester.
The study found that course engagement was more like to occur in small classes, discussion classes, and classes where the instructor was (a) responsive to student questions, (b) assigned effective aids for learning, (c) encouraged students to seek assistance and (d) knew most of the students’ names. In addition we found that emotional engagement and participation engagement were greater for intrinsically motivated students.
These results suggest practical methods for meeting a variety of student needs that have been shown to increase engagement, including students’ need for relatedness - by encouraging them to seek assistance and knowing their names, competence - by assigning effective learning aids, and autonomy - by encouraging intrinsically motivated activities.
back to top
_________________________________________
Assessment Awards Luncheon
The Assessment Awards Luncheon was held on Thursday, February 21st in the NSU Cedar Room. Faculty members who prepare department and program reports were invited to attend and were recognized for their outstanding work on assessment over the last year. The following awards were presented to departments and individuals for their exemplary contributions to assessment at UNK.
Award of Appreciation
Glen Powell.jpg "Finnie Picture")
Research on Assessment
School Psychology * Psychology *Economics* Marketing
* ITEC *eCampus * College of Education
Use of Assessment Data
ITEC * Gifted * History * Psychology * Criminal Justice
* Modern Languages * Women’s Studies
Most Improved Assessment Reporting
Art/Art History
General Studies Assessment
Sociology * Women’s Studies * Psychology * English Literature
* Communications * Economics
Outstanding Scholarship on Assessment
Jean Mandernach
Exemplary Contributions to Assessment
eCampus
back to top
_________________________________________
.jpg "ncalogo")
UNK Focused Visit Self-Study
On April 28-29, a team from the Higher Learning Commission (HLC) of the North Central Association (NCA) will visit UNK to evaluate our progress in assessment of student learning. In the 2004 NCA feedback report, the evaluation team indicated that the assessment process at UNK did not meet NCA requirements and should be revisited after four years to gauge progress toward creating a coherent and sustainable assessment process. Content for the self-study was written by the Director of eCampus, the Director of General Studies, and the Director of Assessment with input from individuals and groups on campus involved in assessment. A draft of the self-study was reviewed by a variety of stakeholders on campus including the SVCAASL, Deans, Faculty Senate, WI/CD committee, GS committee, and the Assessment committee.
The self-study includes chapters on each of the nine areas that NCA called out in their report from their 2004 visit. The document will be sent to HLC at the end of February for their review and feedback before their visit in April. The self-study is available on the assessment website:(http://www.unk.edu/academicaffairs/assessment/index.php?id=31686).
back to top
_________________________________________
In January, UNK officially went “into production” with our new online assessment management system WEAVEonline 4.0. The Office of Assessment staff has begun entering 2006-2007 department/program plans into WEAVEonline. We hope to have all plans entered by the end of the semester. This will provide departments with content for their 2007-2008 assessment reports.
Next year, all departments/programs will be able to use WEAVEonline for annual reporting of their assessment data. Office of Assessment personnel will begin training WEAVEonline users (e.g. department chairs, report preparers) as they are ready to begin inputting their assessment data in preparation for the October 1st reporting deadline. If any department would like to meet this semester to learn more about WEAVEonline and to get an early start on their report, please call Sarah von Schrader, the Coordinator of Assessment to set up an appointment (x8495).
back to top
_________________________________________
Student Assessment Committee Update
In the fall of 2007, the Student Assessment Committee welcomed new members Jessy Hansen, April Becker, Sara Chu, and Tim Davis. The Committee focused on dissemination of the results from the Student Involvement Survey and the National Survey of Student Engagement (NSSE) during the fall semester. The members created posters that were hung in the Student Union and in the residents halls. An article was published in the Antelope Newspaper on the results of the surveys along with information about the Student Assessment Committee.
The Student Assessment Committee is currently developing a survey of student preferences related to instructional practices. The Committee decided to use an instructional practice survey developed at Boise State University (BSU) as a staring point. The BSU survey was developed in 1997; the committee will revise the survey to make it more appropriate for UNK in 2007, e.g., including items related to the use of technology. The survey will ask students about a wide range of instructional practices such as: group work, regular quizzes, lecture outlines posted online, and access to faculty outside of class. Student respondents will be asked which instructional practices they prefer and how often these practices occur. The committee hopes that the survey will provide an opportunity for students to provide feedback about the instructional practices they find most beneficial to learning. The survey will be administered shortly after spring break.
back to top
_________________________________________
.jpg "CAAP logo(1)")
CAAP Writing and Critical Thinking Results
In October of 2007, The CAAP Critical Thinking and Writing tests were administered to a sample of UNK freshmen and seniors. The summary results of the test were returned this January. The CAAP assessment results allow us to compare UNK freshmen and seniors to national normative groups.
Prior to making interpretations based on the data, a few limitations should be clarified. It is not possible to make value-added statements based on our CAAP data. The data are limited in this regard by the fact that we do not have the same (or matched) students taking the tests in their freshman and senior years. Also, the sampling of students for CAAP testing was not random. It was done by selecting classrooms using random methods. If a comparison of means for freshmen and seniors were to be done (using a cross sectional method), one should also be aware from freshman to senior year the composition of the student body changes. For example, the freshman to sophomore retention rate is about 80%. Presumably, many of the freshmen who do not return are not high achievers. If one wanted to make value-added interpretations based on the CAAP data, we could administer the CAAP to the same students during their freshman and senior years. Another option would be to link ACT scores to CAAP scores to evaluate growth.
Although our sampling method of was not ideal, our sample does appear to accurately reflect UNK freshmen and seniors. Demographic characteristics of the UNK sample such as race/ethnicity, age, and gender are similar to those of the UNK undergraduate student body.
Critical Thinking -FRESHMEN:
One hundred and ninety eight (198) freshmen took the CAAP Critical Thinking (CT) test, the average scale score was 60(reference the Institutional Summary Report – Freshmen). A total of 161 (81%) students who took the CT test reported their ethnicity as White/Caucasian. These students had higher scores (mean scale score of 61) compared to Mexican American/Chicano (mean 57) and Asian/Pacific Islanders (mean 53). However, basing interpretations on these numbers is tenuous at best because of the low numbers of non-white/Caucasian participants. If more reliable information is desired, we could over-sample from these groups in future administrations. There was a seven point scale score difference between students who reported English as their first language (mean 61) and those who did not report English as their first language (mean 54). Note that students were not allowed to use translators during the exam. As expected there was an increase of CT scores as reported GPAs increased. Scores were similar across reported majors.
Of the 138 students who responded to the item about how hard they tried on the CT test, most (92%) reported that they “tried their best” or “gave moderate effort”. This is a very high percentage given that students are expected to be less motivated on an assessment like this. In future administrations, we should make additional efforts to be sure that students respond to this item. As the instructions to complete this item are given after the test is over.
The mean scale score for the sample of 198 UNK freshmen was 60. The average freshman at UNK received a critical thinking score equal to or higher than that of 49% of the freshman in the national norm group (N=16,013).
18% of UNK freshmen scored above the 75th percentile nationally,
27% of UNK freshmen scored between the 50th to 75th percentiles nationally,
32% of UNK freshmen scored between the 25th to 50th percentiles nationally,
23% of UNK freshmen scored at or below the 25th percentile nationally.
Critical Thinking (CT) -SENIORS:
One hundred and sixty nine (169) UNK seniors took the CAAP Critical Thinking (CT) test, the average scale score was 63(Reference the Institutional Summary Report – Seniors). A total of 136 (80%) students who took the CT test reported their ethnicity as White/Caucasian. These students had higher scores (mean scale score of 64) compared to Mexican American/Chicano (mean 60) and Asian/Pacific Islanders (mean 56). However, again, basing interpretations on these numbers should be done cautiously, if at all, because of the low numbers of non-white/Caucasian participants. There was a six-point scale score difference between students who reported English as their first language (mean 63) and those who did not report English as their first language (mean 57). Again there was an increase of CT scores as reported GPAs increased. Scores were similar across reported majors, although Social Science majors had somewhat higher scores (mean 65) than other reported majors.
Of the 128 students who responded to the item about how hard they tried on the CT test, most (85%) reported that they “tried their best” or “gave moderate effort”. This is a fairly high percentage given that students are expected to be less motivated on an assessment like this. We should make additional efforts to be sure that students respond to this item.
The average scale score for the sample of 169 UNK seniors was 63. The average senior at UNK receiv national ed a critical thinking score equal to or higher than that of 50% of the seniors in the norm group (N=10,547).
25% of UNK seniors scored above the 75th percentile nationally,
26% of UNK seniors scored between the 50th to 75th percentiles nationally,
25% of UNK seniors scored between the 25th to 50th percentiles nationally,
24% of UNK seniors scored at or below the 25th percentile nationally.
Writing – FRESHMEN
There is less information about the demographic characteristics of the students who took the writing test. It is not possible to compare scores by demographic groups using the CAAP writing score report. UNK freshmen (N=196) had an average scale score of 2.8, while the national normative group (N= 11,886) mean was 3.1 (reference the Institutional Summary Report – Freshmen). The following lines compare the frequency distribution of the UNK freshmen to the freshman normative group.
18% of UNK freshmen scored above the 75th percentile nationally,
40% of UNK freshmen scored between the 50th to 75th percentiles nationally,
5% of UNK freshmen scored between the 25th to 50th percentiles nationally,
37% of UNK freshmen scored at or below the 25th percentile nationally.
Writing – SENIORS
UNK seniors (N=191) had an average scale score of 3.0, while the national normative group (N= 6489) mean was 3.3 (Reference the Institutional Summary Report – Seniors). The following lines compare the frequency distribution of the UNK seniors to seniors in the national norm group. The average senior at UNK received a writing score equal to or higher than that of 41% of the seniors in the senior normative group.
14% of UNK seniors scored above the 75th percentile nationally,
27% of UNK seniors scored between the 50th to 75th percentiles nationally,
26% of UNK seniors scored between the 25th to 50th percentiles nationally,
33% of UNK seniors scored at or below the 25th percentile nationally.
Some Summary Observations
On both tests the percentage of seniors who feel into the lowest quartile is perhaps an area for concern. If this measure is used in the future it will be of interest to monitor the relative size of this group. This will be of particular interest if there are new initiatives targeting the writing and critical thinking skills of UNK students. ACT suggests using the CAAP data to set benchmarks. For example, at the next administration of this measure we would hope to see fewer students falling into the lowest quartile nationally. On the Critical Thinking test, 23% of UNK freshmen scored in the lowest quartile; 24% of UNK seniors scored in the lowest quartile. On the writing test, 37% of UNK freshmen fell into the lowest quartile nationally; this low achieving group is an important target for intervention. There are several initiatives at UNK designed to improve the writing skills of UNK students over the course of their college career (General studies program, WI program, Composition program). Among seniors, the percent in the lowest national quartile is 33%; this number is certainly higher than would be desired. ACT also suggests examining the course-taking information for students who fall in the lower quartiles to determine if an intervention is warranted. To discover the actual students who fall in this lowest quartile, we could look at the CAAP student roster.
The CAAP institutional reports are available on the assessment website. The normative information in these reports is for sophomore students, and should not be used when making interpretations. Instead ACT provided normative information for freshman and senior who took the CAAP in their User Norms 2007-2008 brochure, these norms were used in the following presentation of data.
back to top
_________________________________________
Upcoming Assessment Conferences
The Office of Assessment can provide funding to faculty who are interested in attending any of the following assessment related conferences.
- Higher Learning Commission
Finding Common Ground: Accreditation, Assessment and Accountability
April 11-15, 2008
Chicago, IL
http://www.ncahlc.org/annualmeeting/
- The 2008 Assessment Institute
October 26-28, 2008
The Westin Indianapolis
Indianapolis, Indiana
Pre-Institute Workshops: October 26, 2008
Institute Dates: October 27-28, 2008
Resources for Higher Education Assessment:
http://www2.acs.ncsu.edu/UPA/assmt/resource.htm
This site, maintained by North Carolina State University Planning and Analysis, is an exhaustive listing of assessment materials available on the Web
back to top
______________________________
|