Chemistry Reports

Report - 2006

Chemistry Assessment Report 2006

Submitted Fall, 2006

UNK Department of Chemistry
2005-06 Assessment Results and Analysis

 


Department Learning Goal #1 -- Upon completing a course required for a Chemistry degree, students will have developed the ability to critically evaluate and solve problems through the application of appropriate chemistry principles.


Measurement Tools and Participants

The learning objectives for Department Learning Goal #1 were measured in two ways:

Standardized national exams (Direct Measure category 1. a. i.) produced by the American Chemical Society (ACS). Participants were all students in the following courses, each of which has an associated subject-specific ACS Exam.

Learning Objective Subject Area Course(s) Associated ACS Standardized Exam
General Chemistry • CHEM 160 - General Chemistry I
• CHEM 161 - General Chemistry II
• ACS 1st-Term General Chemistry Exam
• ACS General Chemistry Exam
Organic Chemistry • CHEM 361 - Organic Chemistry II • ACS Organic Chemistry Exam
Analytical Chemistry • CHEM 301 - Techniques of Chemical Analysis
• CHEM 475 - Instrumental Analysis
• ACS Analytical Chem. Exam
• ACS Instrumental Analysis Exam
Physical Chemistry • CHEM 480 - Physical Chemistry I
• CHEM 481 - Physical Chemistry II
• ACS Quantum Exam
• ACS Thermo & Dynamics Exam

Direct measure1. a. ii. – local final exams – were used for assessing academic achievement in CHEM 145 (Introductory Chemistry) and 150 (Introduction to Organic & Biochemistry). Participants were all students in these courses. Instructors divided their final exam questions into sections corresponding to material covered in the course. While individual exam questions may vary slightly from year-to-year, the types of exam sections and relative weighting will remain consistent.

Benchmark criteria for successful achievement:

  • Standardized national exams -- student exam scores will average at or above the 50th percentile (the national average)
  • CHEM 145, 150 Exams -- The average score on each exam section for each course will be no lower than 70%. If this benchmark is not met in three successive years, the format and delivery of the course will be reviewed and modified as appropriate.
Learning Objective Subject Area Courses ACS Exam Average
(percentile)
Meets Criterion?
General Chemistry CHEM 160
CHEM 161

50
51

yes
yes
Organic Chemistry CHEM 361 59 yes
Analytical Chemistry CHEM 301
CHEM 475
68
not offered in 06-07
yes
--
Physical Chemistry CHEM 480
CHEM 481
61
60
yes
yes

Last year, CHEM 160 failed to meet the benchmark criterion. This year, it met the minimum score to meet that criterion (50%tile average). The relatively low scores of the CHEM 160 and CHEM 161 courses indicate a potential trend. First semester chemistry (CHEM 160) only requires basic mathematics skills. The prerequisites for subsequent chemistry courses (C or better grade in the previous chemistry course) ensure that better prepared students continue in the major courses. Thus, the average percentiles in ACS final exams from freshman (CHEM 160, CHEM 161) to sophomore (CHEM 361) to junior and senior level (remaining courses) should increase.

In the fall semester of 2006, a new textbook was chosen for students in the CHEM 160-161 sequence. To further explore the effects of the new textbook on student learning, details of student responses in the topic areas for the CHEM 160 and 161 exams were tabulated (for the 2005-2006 academic year). These
 

CHEM 160-ACS Exam (2002 version) 

CHEM 161-ACS Exam (2001 version) 
Categories  % students with correct answer 

Categories 

% students with correct answer 
Chemical Foundations 58

Solutions

50
Atomic Theory & Mole 61

Kinetics

64
Stoichiometry 52

Equilibrium

38
Chemical Reactions 61

Acids & Bases

35
Gases 55

Electrochemistry

54
Thermochemistry 57

Descriptive & Nuclear

53
Structure & Periodicity 68    
Bonding & Structure 64    
Orbitals 64    
Liquids & Solids 50    

results will be compared to the 2006-2007 breakdown to assess the strengths and weaknesses of the new text. While a specific detailed review has not been triggered, the faculty teaching these courses are currently examining these topic areas as potential topics that need further (or different) emphasis in the course in an effort to improve the student level of understanding to match the national norm (50th %tile). Low scores in the ACS exam for CHEM 161 (equilibrium and acid-base chemistry) are indicative of the area of CHEM 161 that is the most math-intensive. The new textbook places greater emphasis on problem solving skills in these subjects and may help students improve in these areas.

The breakdown and examination of final exam grades for CHEM 145 and CHEM 150 was not undertaken in 2005-2006. The assessment of CHEM 145 and CHEM 150 final exam grades will be resumed in 2006-2007. Low scores in any of those topics in the 2006-2007 academic year will trigger a more detailed review.


Department Learning Goal #2 -- Chemistry graduates will be able to coherently organize and present chemical information through written and oral presentation.

Measurement Tools and Participants

The learning objectives were measured using an evaluation of the final oral presentations – direct measure category 1. b. x. -- given by all students enrolled in CHEM 469 (Chemistry Seminar).  Because all chemistry majors are required to complete this course, this measure evaluates all chemistry graduates.  A copy of the evaluation form is linked.

Benchmark criteria for successful achievement

  • Minimum Proficiency for All Chemistry Graduates – The median evaluation question score for each student will be no lower than 3 points on the 5-point scale.
  • Desired Proficiency for Best-Performing Students -- The median evaluation question score for each student will be no lower than 4 points on the 5-point scale.

Results and Discussion

Twelve students completed CHEM 469 (Chemistry Seminar) in Fall 2005.  Of those students, one did not complete the course and received a failing grade.  The remaining eleven students completed the presentations, review article, and other coursework and were graded.  The results of their presentations over the review article they constructed are summarized below:

 

Mean Presentation Score

# Scores below 3

Presentation Organization

4.2

0

Effectiveness of Communication

3.9

1

Effectiveness of Illustrations

4.1

1

Understanding of Chemical Principles

5.0

0

Appropriate Level for Audience

4.2

0

Analysis of Themes from Multiple Sources

4.3

0

Depth of Analysis

4.0

1

 

 

 

Median for Class

4.4

 

One student did not meet the minimum and desired proficiency criteria for the course (see “# Scores below 3” column in table).  This student was an ESL student that struggled with English grammar.  Additional assistance in grammar, spelling, sentence structure and layout were offered to the student to improve their abilities. The remaining students did meet the minimum and desired proficiency level.  The assessment of the data supports the grades given to all twelve students.

Note on Portfolios 

A second direct measure, 1. b. xii. – portfolios – continues to be in preparation.  Files have been established for all declared chemistry majors and are continually updated.  From each applicable chemistry course CHEM 361L, CHEM 480, and CHEM 469), at least one sample of writing is placed in each student’s file.  Prior to graduation, each student’s materials will be evaluated for improvement in organization, clarity, effectiveness of illustrations (when applicable), and understanding and application of chemistry principles.  This evaluation will not commence until the 2006-07 academic year, when 4-year portfolios will first be available for graduating seniors. 

 


Department Learning Goal #3 -- Chemistry graduates will have the necessary skills and knowledge to acquire entry-level positions in the field (including industry and high school teaching) or for admission to their graduate or professional schools in the field of their choice.


and

 


Department Learning Goal #4 -- Chemistry graduates will value how their education enables them to make more informed assessments of and take actions on chemistry-related health and environmental issues.


Measurement Tools and Participants

All of the learning objectives for Department Learning Goal #3 and most for Department Learning Goal #4 were measured using surveys of graduating seniors – indirect measure category 2. iii.  The survey was developed by the UNK Department of Chemistry and is linked.

Benchmark criteria for successful achievement

  • At least 75% of chemistry graduates who choose to pursue a chemistry-related profession will acquire employment with the company of their choice or gain admission to the graduate or professional school of their choice
  • All scores for questions pertaining to the value of chemistry course content and course activities/resources in the graduates’ learning of chemistry will be at least 3 (on the 5-point scale).
  • All scores for questions pertaining to the recommendation of attending UNK and being a chemistry major – “If you were starting your college career over, would you attend UNK?” and if so, “. . . would you be a chemistry major?” – will be at least 3 (on the 5-point scale).
  • Average value-question scores from surveys received from 4-year alumni will be at or above the average from the surveys they had completed just before graduation.

Results and Discussion

Each of the ten chemistry majors that graduated in 2005-2006 indicated responses on the survey.  Of those students, nine (90%) gained employment or admission in their chosen field.  The tenth student was, at the time of the survey, still waiting on a response to their application for their chosen field.  Since that survey, the student did gain admission to their desired program of study as a professional school student.  This category meets the benchmark criterion.

From the survey, questions pertaining to the value of chemistry course content, course activities/resources, and recommendation of attending UNK and being a chemistry major, most ratings were very positive, but some scores below 3 (out of the 5-point scale) were noted.  The specific data has been collected into the table below.

As was mentioned in the previous assessment report (2004-2005), “an occasional single low response can often be attributed to a student perceiving that his or her career aspiration poorly matched the particular area of chemistry.”  Multiple and continued low responses in the same category, on the other hand, can indicate a trend that needs to be examined.  At this point, it appears that some trends are beginning to be observed.  For instance, Physical Chemistry has (for the last two years) received multiple low scores in each category of the assessment tool.  Moreover, the lowest average ratings in each category are given to Physical Chemistry, a course

  # Below 3 
Rating for Preparation for Profession   
Physical Chemistry 

3

 

 

Rating for Understanding of Chemistry 

 

Biochemistry 

2

Physical Chemistry

2

 

 

Rating for Enhancing Informed Decision-Making on Health/Environmental Issues 

 

Organic Chemistry

1

Biochemistry 

1

Physical Chemistry 

3

Inorganic Chemistry 

1

 

 

Rating for Value of Department Resource or Activity 

 

Research

1

Library Journal Holdings

4

Library Book Holdings

7

ACS Student Affiliate

2

that underwent a major curricular change in the 2004-2005 academic year.  If this trend in Physical Chemistry and Biochemistry continues in the surveys of the 2006-2007 graduating class, a close examination of the courses will be undertaken by the department.  If such an examination is made, possible recommendations for curricular or pedagogical changes will be proffered.  In the meantime, the department will  make greater efforts to point out the relevance and importance of Physical Chemistry and Biochemistry to student majors. 

Additional low responses were noted in the students’ perceived value of the library holdings.  Unlike last year’s survey, there were multiple poor responses (the lowest overall score occurred in the library book holdings area) for the library.  Based on this data, it appears that students do not place much value in the library as a physical entity.  However, many of the students do use the on-line resources offered by the library (as this is a requirement in at least one course in the major).  An aim to correct the perception (and to further stress that on-line resources are offered by the library) will be made in next year’s survey, and a separate question will be considered in order to tease out the actual value students assign to the different resources offered by the UNK Library.  This is further discussed in the Assessment of the Assessment section.

The low response to the value of the ACS Student Affiliate may correlate directly with a student’s level of activity in the club.  More is discussed in the Assessment of the Assessment section.

Additional Objective and Measurement for Department Learning Goal #4

As was mentioned in the 2005-2006 Assessment Report, the value perceptions of General Studies student are important to the Chemistry program, given the fact that such courses aid in the preparation of students with a liberal arts background.  To that end, the Chemistry Department has implemented an additional learning objective related to Learning Goal #4.  Initial data on this learning objective was reported in the 2005-2006 Assessment Report.  That learning objective is…


Department Learning Goal #4b-- Chemistry students in General Studies courses will value the learning of their chemistry course content for the understanding of chemistry as a scientific discipline and enabling them to make more informed decisions on public issues of a scientific nature.


Measurement Tools and Participants

Indirect measure category 2.iii. – surveys – was used for this assessment measurement.  Questions are derived from the Chemistry Department graduate survey and the UNK General Studies objectives.  This short survey was administered in every Chemistry General Studies course (CHEM 145, 150, 160, 161) along with the teaching evaluation.

Benchmark criterion for successful achievement

Each year for each General Studies course, the average survey response for each question will be no lower than 3.5.  If this benchmark is not met in three successive years, the General Studies course format and delivery will be reviewed and modified as appropriate.

Results and Discussion

Results from the surveys administered during the 2005-2006 academic year are tabulated below.  The data reflect survey results from multiple sections and multiple semesters (no correlation was found to indicate a difference in student responses from section to section, or from semester to semester.)  No data was collected in CHEM 150 in the 2005-2006 academic year.  It will be collected for use in the next assessment report.

CHEM 145 – Intro to Chemistry   
Use critical thinking, reasoning, and analyzing in solving problems.  3.71
Apply scientific methodology in a laboratory setting.  3.85
Comprehend how scientific concepts originate and are validated and refined. 3.68
Use specialized vocabulary to understand matter and energy.  3.68
Make more informed decisions about public issues pertaining to science.  3.65

CHEM 160 – General Chemistry I 

CHEM 161 – General Chemistry II 

Use critical thinking, reasoning, and analyzing in solving problems.

3.32

Use critical thinking, reasoning, and analyzing in solving problems.

4.03

Apply scientific methodology in a laboratory setting.

3.37

Apply scientific methodology in a laboratory setting.

4.02

Comprehend how scientific concepts originate and are validated and refined.

3.35

Comprehend how scientific concepts originate and are validated and refined.

3.93

Use specialized vocabulary to understand matter and energy.

3.49

Use specialized vocabulary to understand matter and energy.

4.09

Make more informed decisions about public issues pertaining to science.

3.15

Make more informed decisions about public issues pertaining to science.

3.58

Students in CHEM 145 and CHEM 161 indicated that they perceived a strong alignment of their course to the general studies objectives.  In each case, their responses exceeded the benchmark criterion for success in this learning goal.  The CHEM 160 student responses, however, are below the 3.5 set as the benchmark criterion in each of the examined categories.  Yet, their responses are above the average value for the measurement (3.0, on a 1-5 scale).  While this is the first year that these data have been collected, the results do indicate that students in the first semester of the majors sequence (CHEM 160) have more difficulty recognizing that the topics in the course are aligned with the overall objectives of a general studies course.  A closer examination of this data, with recommended adjustments to the instruction and/or curriculum for that course, will be triggered if the low responses continue over the next two years. 

In the 2006-2007 academic year, the department changed the textbook used in CHEM 160 and CHEM 161.  The textbook that was chosen is written to include more applications of chemistry he new textbook may help address the low perception of the students for the applicability of CHEM 160 to basic general studies requirements.  Further efforts will be made by the faculty to help students make the connections between the general studies objectives and the objectives of CHEM 160.

 


Department Learning Goal #5 -- Chemistry students will have developed the ability to use proper scientific methodology and laboratory technique in the investigation of chemical principles through experiment.


Measurement Tools and Participants

Two measurement tools were used for assessment pertaining to this learning goal. All General Chemistry (CHEM 160 & 161) students were assessed

Direct measure category 1. a. i. – national exams – were used to assess laboratory content understanding. Laboratory questions are a part of the ACS standardized exams given in General Chemistry courses (CHEM 160, 161).

Direct measure category 1. b. xiii. – projects – were used to directly evaluate General Chemistry student (CHEM 160, 161) laboratory skills. As proper technique directly relates to the quality of experiment results, the results from one quantitative analysis experiment in each course will be assessed.

Benchmark criteria for successful achievement

Laboratory Exam Questions. Each year, the average laboratory question section score will correspond to the national exam average or better (50% correct on the latest CHEM160 exam and 50% correct on the latest CHEM 161 exam). If this benchmark is not met in three successive years, the General Chemistry laboratory format and delivery will be reviewed and modified as appropriate.

Laboratory Projects. As mentioned in the 2004-2005 report, the laboratory projects benchmark was adjusted to report skills associated with the laboratory practical exam (see the Assessment of the Assessment section for details). Each semester, the desired proficiency for the General Chemistry courses (CHEM 160L and CHEM 161L) will be an average laboratory practical exam score greater than 70%. If the 70% benchmark is not met in three successive years, the General Chemistry laboratory format and delivery will be reviewed and modified as appropriate. For this, the first year of collection of lab practical data, a single section of both CHEM 160L and CHEM 161L were used for this measure. In the 2006-2007 report, data from all sections of these courses will be used in this measure.

Results and Discussion

The table below summarizes results from both assessment tools:

Laboratory Exam Questions % Correct Benchmark Criterion (%) Met Criterion?
CHEM 160 54 50 yes
CHEM 161 67 50 yes
 
Laboratory Project % Correct Benchmark Criterion (%) Met Criterion?
CHEM 160L 73 70 yes
CHEM 161L 74 70 yes

Assessment of Assessment

Existing Goals, Objectives, Measurement Tools, and Criteria

Based on the 2005-06 assessment results, the existing learning goals, objectives, and measurement tools successfully serve the department’s needs. The following are issues of concern that have arisen.

Addition of question to Graduate Survey.  As described in the 2003-04 and 2004-05 reports, it was intended to have the following question added to the graduate survey. Mistakenly, it has not yet been added; however, the question has been added to the 2006-07 surveys:

What was your level of activity in the department Chemistry Club? (circle one)

Very Active                 Somewhat Active                   Not Active

Additionally, a question regarding the usefulness of the library and its chemistry holdings has been raised from the data collected in the previous surveys (especially in the 2004-2005 survey).  To determine the exact problem students report with the library, the following questions will be added to the 2006-2007 surveys:

How often did you use the chemistry books in the library? (circle one)     
  Often Occasionally Seldom Never
         
How often did you use the chemistry journals in the library? (circle one)
  Often Occasionally Seldom Never
         
How often did you use the chemistry journals from the internet? (circle one)
  Often Occasionally Seldom Never

Laboratory Practical Exams.  In 2005-2006, the department established a laboratory practical exam in CHEM 145, CHEM 160L, and CHEM 161L.  This practical exam requires the student to use available equipment properly to measure particular quantities or concentrations.  Students are exposed to this equipment throughout the semester and routinely use the techniques to solve problems within the normal practice of the laboratory course.  The assessment of student understanding and ability in these general studies laboratory courses can be more accurately and thoroughly examined using the laboratory practical exam.  This assessment report indicates the first examination of the data.

Potential Problems with 4-Year Portfolios.  It has become apparent that many students do not declare chemistry majors until their sophomore or junior year.  This limits the number of writing samples – and the timeframe of those samples – that can be collected for the portfolio assessment.  In 2006-07, the first year that assessment data in this area will be collected, the impact of this potential problem will be noted.

Instead of a comprehensive 4-year portfolio, focus on this assessed part of the student career will center on the collection of specific writing samples from the students during their sophomore and senior years (in the courses: CHEM 361L, CHEM 480, and CHEM 469).  One written assignment from each of these courses will be collected from declared chemistry majors, added to the portfolio, and graded using the rubric outlined in the department’s Writing Intensive guidelines document.
 

Undergraduate Research

Collaborative student/faculty research has become an increasingly essential component in the undergraduate chemistry curriculum delivery.  There are a number of positive outcomes that are often discussed in anecdotal and qualitative fashions.  Development of meaningful assessment tools has been an issue of national debate due to issues such as complicated control factors, small student numbers (per discipline), and the great breadth and paradigm variety among academic disciplines.  The department is interested in exploring the development of undergraduate research assessment in cooperation with the Coordinator of Assessment.

General Studies

The General Studies objectives over which CHEM 145, 150, 160, and 161 student are surveyed could be said to establish minimum levels of exposure to and proficiencies in Chemistry content and methodology for all UNK students.  True fulfillment of these objectives would be most accurately assessed using data from non-science majors.  Unfortunately, this audience makes up a disproportionately small portion of chemistry General Studies classes.

All UNK students must choose science courses from two out of four science departments –at least one of the two courses must include a laboratory component.  Initial data collected in 2002-2003 appeared to support this fact.  Recent data, from the registrar’s office for Fall 2005, supports the statement that very few students choose a chemistry course as a General Studies elective.

Furthermore, data from Fall 2005 and Spring 2006 illustrate that the majority of the enrollment in GS courses in ALL other disciplines in the College of NSS (except Chemistry) is made up of students that are satisfying a general studies requirement (not shown).  In fact, ~80% of all students in the other programs within CNSS (BIOL, CSIS, CJUS, GEOG, HIST, MATH, PHYS, PSCI, PSY, SOC, SOWK, and WSTD) take a department’s GS course to fulfill a general studies requirement. The Physics Department is the next closest in the percent of students taking GS courses as a requirement.

Fall 2005 - Major Distribution in Natural Science General Studies Courses 

  BIOL  CHEM  PHYS  GEOG 
% Gen. Studies Courses*  79.0 38.8  64.8 91.4
% Elective course for major**  2.7 0.0 0.5 0.3
% Required for major*  18.2  61.2   34.7 8.3
TOTAL  100  100  100 100
*percent of students taking a department’s GS course to satisfy a general studies requirement only    
**percent of students taking a department’s GS course as an elective within their major program    
***percent of students taking a department’s GS course meet a requirement for their major    
 
With such a small percentage of non-science majors (who come from the General Studies students group) in chemistry General Studies courses, data collected by the Chemistry Department for General Studies assessment is of very limited use.  If administrative steps were taken to equalize non-science major enrollment across the four natural science departments, much more meaningful General Studies assessment could be conducted.