Submitted Fall, 2006
2006 Assessment Report
Department of Industrial Technology
B.S. Aviation Systems Management (ASM)
B.S. Construction Management (CM)
B.S. Industrial Distribution (ID)
B.S. Telecommunications Management (TM)
Graduates with a degree from the Department of Industrial Technology at the University of Nebraska at Kearney (UNK) will demonstrate the knowledge, skills, and values commensurate with the competencies identified for a professional in Industrial Technology with the ability to orally and visually communicate these competencies to their constituencies.
The successful student will develop:
- knowledge of the applications of technology as they pertain to the industrial sector of the economy, specifically aviation systems, building construction systems, industrial distribution systems, and telecommunication systems.
- knowledge of the scientific and mathematical concepts that form the basis of technological concepts.
- knowledge of the equipment, materials, and processes that comprise the end product or service of these systems.
- knowledge in the manipulations of data as it impacts the technological and economic implications of the system.
- skill in the safe and efficient manipulation of the technical equipment associated with the requisite activities within the systems.
- skill in the ability to organize and present the technical concepts of the discipline through written and oral communication.
- skill in developing working solutions to problems typified by the discipline.
- appreciation of values pertaining to work, education and the community.
The Industrial Technology (ITEC) department assesses its program effectiveness in keeping with the National Association of Industrial Technology (NAIT) accreditation standards. All programs in the department are currently accredited by NAIT with three programs having being reaccredited in November 2006 for an additional six year period. In addition, during November 2006, the Aviation Systems Management program received its initial accreditation for a six year period putting all four department programs on the same accreditation schedule. Assessment measures used for all four ITEC programs include both direct and indirect measures. Direct measures include the Final Evaluation of Intern by Work Site Supervisor (not applicable to Aviation Systems Management) and the Comprehensive Exam for each program. Indirect measures include the Employer Survey, the Graduate Survey and the Student Confidence Scale. The Student Confidence Scale correlates directly with the Student Outcome Matrix for each program. The data collected from each assessment instrument is evaluated by the ITEC faculty and staff, work site industry representatives and the ITEC program advisory committees as appropriate to determine what – if any – changes may be required in the curriculum.
Final Evaluation of Intern by Work Site Supervisor: All CM, ID, and TM students are required to participate in an internship and this evaluation measure is a required component of the internship. The intern's supervisor completes the evaluation at the end of 480 hours of work by the intern. The information is discussed with the intern and both parties sign the evaluation. The purpose is to document progress since the midterm evaluation and evaluate the training plan objectives set at the beginning of the internship as well as work performance. The information gathered through this instrument is summarized by degree program and distributed to faculty for review. It is used to determine what – if any – changes may be required in the curriculum.
Comprehensive Exam: A comprehensive exam for each program was prepared during the 2005-2006 academic year and first administered during the Spring 2006 semester. This exam was administered to all graduating seniors to determine the knowledge level mastered, both technical and non-technical, during the students UNK experience. Feedback from this instrument will be used to determine what – if any – changes may be required in the curriculum. As the exam was administered only once it is still too early to make meaningful decisions based upon one set of exam results.
Employer Survey: For every student returning the Graduate Survey (see below) an Employer Survey is sent to the graduate’s current employer. Employers are asked to provide information about the graduate in terms of their initial hire position, current position, the degree to which they have increased responsibility, work performance, productivity, business techniques, personal characteristics, and the employers overall satisfaction with the graduate. Feedback from this instrument is used to determine what – if any – changes may be required in the curriculum.
Graduate Survey: The Graduate Survey is sent to all graduates one and five years after graduation. The purpose of the survey is to determine the success of graduates in the workplace. Information is sought about how satisfied the graduates are with the quality of instruction received, quality of facilities, equipment and academic services. Employment status, salary and advancement information is also sought. Feedback from this instrument is used to determine what – if any – changes may be required in the curriculum.
Student Confidence Scale: The Department of Industrial Technology prepares students for successful careers in business and industry. The Student Confidence Survey is administered just prior to graduation and helps curriculum planners within the department gain an understanding as to how confident the students feel that they are prepared in terms of knowledge, skills and abilities to successfully enter the workplace upon graduation. The Student Confidence Scale correlates directly with the Student Outcome Matrix for each program. Feedback from this instrument is used to determine what – if any – changes may be required in the curriculum.
Indirect Assessment Instruments:
- Graduate Survey – ASM, CM, ID, TM
- Employer Survey – ASM, CM, ID, TM
- Student Confidence Scale – ASM, CM, ID, TM
Direct Assessment Instruments:
- Final Evaluation of Intern by Work Site Supervisor – CM, ID, TM
- Aviation Systems Management Comprehensive Exam - ASM
- Construction Management Comprehensive Exam - CM
- Industrial Distribution Comprehensive Exam - ID
- Telecommunications Management Comprehensive Exam – TM
The Final Evaluation of Intern by Work Site Supervisor has been in use since 1994 in various forms. The “Comprehensive Exam” direct measure was first administered during the Spring 2006 semester.
Aviation Systems Management (ASM) - 2006
The ASM program assessment activities were expanded and refined for the 2005 - 2006 academic year. The 2006 aviation program graduates completed a confidence survey of the ASM Advisory Committee approved program competencies while the 2000 and 2005 graduates were asked to complete the Graduate survey. These instruments, as well as the newly implemented Comprehensive Exam and the Employer Survey, form the core of the ASM assessment effort.
It must be noted that the competencies used for this survey were developed for the current ASM program but administered to the graduates of the former Airway Science program. It is therefore recognized that large anomalies will exist because many of the core competencies being measured were never taught to this group of graduates. Thus, the validity of the assessment will be compromised. However, experience in implementing the assessment process at this time will prove beneficial when the first ASM graduating class participates in the assessment process in the Spring of 2008.
ASM - Assessment Process Evaluation:
The ASM faculty has discussed the existing assessment process and developed suggestions for making improvements. The revisions to each assessment instrument are discussed individually following each assessment instrument analysis.
ASM - Graduate Survey Analysis:
The ASM Graduate Survey is administered electronically to aviation program graduates who have been out of the program for one and five years and is intended to give those individual(s) a conduit to provide feedback to the program after they have been in industry. The ASM Graduate Survey is in reality a subset of the general ITEC Graduate Survey and the questions asked do not all pertain exclusively to the ASM program and faculty.
The graduate survey had a single respondent, 1 out of 5, which makes it very difficult to draw meaningful conclusions. However, in general term, the feedback was positive and there was overall satisfaction with UNK. While the overall satisfaction appears good, there were specific areas that indicated a need for improvement.
When reviewing the specific questions in the graduate survey, the respondent gave a “dissatisfied” reply to three out of the twenty questions asked. Detailed analysis of these specific possible problem areas is as follows:
- When asked about the respondent’s satisfaction with the recreational / athletic / cultural services provided by UNK, a reply of “dissatisfied” was recorded. Given there was no additional comment and that the majority of the experiences related to this topic are outside the control of the ASM program, no immediate action is contemplated.
- When asked about Career Planning and Job Resources, the respondent once again replied with “dissatisfied”. This is viewed as a legitimate problem within the ASM program and steps have been taken to strengthen this area. As addressed in the 2005 Assessment report, the ITEC-171 class has been altered to include specific information about aviation career planning and all students in the ASM program are also required to take ITEC-110 which has, as a primary purpose, career planning information. Additionally, UNK has implemented required 4 year advising. The intent of the extended advising requirement is to provide career planning support during the Junior and Senior years. It is anticipated that this area will continue to show some dissatisfaction until the first graduates of the current ASM program take this survey.
- The final area marked as “dissatisfied” was program advisement. As discussed in the 2005 assessment report, this group of students came from an era when UNK only required academic advising during the freshman and sophomore years. This policy has been changed and now all UNK students must receive academic advising every semester before they can register for classes. It is anticipated that this change will address the overall satisfaction within this area.
When compared with the data from the 2005 graduate survey, only three categories, as compared to ten last year, rated a “dissatisfied” response. It could be said that significant improvement has been made, however, the authors believe that those changes made as the result of the 2005 graduate survey would have no influence on the respondent of the 2006 version and thus relate the decline in “dissatisfied” responses to mere chance.
ASM - Graduate Survey Suggestions:
The ASM faculty will suggest the following modifications to the ITEC faculty for incorporation into the 2007 Graduate Survey. These suggestions are intended to make the Graduate Survey a more productive instrument in the assessment process.
- Question 8 – “Based on your experience at UNK how satisfied were you with each of the following student services?”
- Include a comment field and solicit feedback as to how to improve areas where they may not have been satisfied.
- Question 9 – “Based on your experience at UNK how satisfied were you with each of the following academic services?”
- The General Studies Courses category might be subdivided to indicate the quality of the instruction and the appropriateness of the course to students program of study.
- The Quality of Course Instruction and Faculty Qualifications could be subdivided between General Studies and Program Specific courses.
ASM - Employer Survey Analysis:
No responses were received from the Employer Survey and thus the only meaningful conclusion that can be reached is that the ASM faculty needs to take a lead role in directly contacting graduate employers to make sure they realize how important this information is to the Aviation Systems Management program.
ASM - Employer Survey Analysis Suggestions:
Once again, the faculty is, in general, satisfied with the employer survey. However, the lack of participation is troubling. It is anticipated that direct contact by ASM program faculty may serve as an impetus for employers to complete the survey. Therefore, the ASM faculty will request that staff advise us when the survey has been sent so we can attempt contact.
ASM - Student Confidence Scale Analysis:
The ASM, Student Confidence Survey was well received by graduating seniors. Though the number of graduates was small, all that were requested to complete the survey instrument did so.
The results of the ASM Student Confidence Scale survey were really quite good. This survey instrument was divided into three sections; 1) Competencies for all ASM graduates; 2) Competencies for Flight Operations emphasis graduates; and 3) Competencies for Aviation Support Services emphasis graduates. The students rated each of the applicable ASM competencies on a scale from one to seven with one being “Insufficient Ability” and seven being “Excellent Ability”. The lowest average response was of 4.0 on one competency and 4.67 on another, all other competencies were rated with an average confidence level of 5.0 or greater with several averaging at the highest level of 7.0.
Of those competencies with individual responses less than 5.0, the majority were in the broad area of “business knowledge”. This result is in line with what the ASM faculty were anticipating as the students completing this survey were gradates from the former Airway Science program that did not have a dedicated business component. The ASM faculty had actually anticipated a much lower confidence level in this area.
Only one competency had a response with level of 3.0 and this was in the area, “Explain the career path options in the aviation field”. The response to this competency echoes that of the graduate survey completed by one of our 2005 graduates. As a result of strengthening both the ITEC-110 and ITEC-171 classes and the 4-year advising requirements, the ASM faculty expects improvements will be demonstrated in the Spring 2008 surveys.
In summary, the ASM faculty is generally pleased with the results of the ASM Student Confidence Scale survey. Some rewording and consolidation of the competencies is probably in order to obtain more useful information for future reference.
ASM - Student Confidence Scale Suggestions:
In general, the ASM faculty is satisfied with the Student Confidence Scale survey. However, it has been suggested by multiple sources that the ASM program competencies are overly comprehensive. Thus, the ASM faculty is currently reviewing the program competencies to determine if logical consolidation can be accomplished. As revisions are formulated, the ASM Advisory Committee will be consulted to ensure the integrity of the current competencies is not compromised.
ASM - Comprehensive Exam Analysis:
The ASM program, as with all programs in the ITEC department, initiated a Comprehensive Exam, direct assessment process, with the 2006 graduates. For the ASM program, this process consisted of two exams, the first being multiple choice questions generated by faculty teaching the ITEC core, technology, business, and common aviation courses; and, the second, consisting of FAA Knowledge Test questions pertinent to those pilot certificate and ratings required by the ASM program. These tests were administered without warning to all ASM 2006 graduates in an attempt to quantify the retention of the materials presented throughout the graduates’ course of study. It must be reiterated that the graduates’ taking the general ITEC direct measures survey had very little of the course preparation required for the current ASM students and as such the result will prove to be essentially useless for valid assessment purposes.
As discussed in the previous paragraph, a large portion of the current Comprehensive Exam is not pertinent to the collegiate experience of 2006 graduates and thus, will not be discussed. However, the last 14 questions on this exam dealt with aviation safety and aviation law, as both subjects are central to ASM competencies and required by all graduates. These courses were also required under the former Airway Science program and therefore the results of these assessment questions should give valid information for program enhancement. The responses to these questions were quite gratifying in that, with two exceptions, all had at least two out of three correct answers. Detailed analysis of those questions having less than two out of three correct answers is as follows:
- None of the three respondents correctly answered the following question:
Basically, the demand management approach to reducing delay depends on the:
- Price mechanism.
- Assigning of quotas.
- FAA.
- Air carriers.
It was determined by the ASM faculty that this question was representative of information all ASM students should know and therefore additional emphasis in determining airport capacity will be included in the next offering of ITEC-392, Airport Management, which is currently scheduled for the Fall semester of 2007.
- Only one out of three respondents correctly answered the following question:
The FAA may issue a letter of correction if:
- The violation did not occur in an operation for compensation or hire.
- The FAA is satisfied that a constructive attitude is present and that remedial training will lead to future compliance.
- You do not have a record of prior non-compliance.
- All of the above.
It was determined by the ASM faculty that this question was representative of information all ASM graduates should know and therefore additional emphasis in determining FAA enforcement actions is being included in the current and future offerings of ITEC-391, Aviation Law.
The second set of Comprehensive Exams, used as a direct measure for the ASM Flight Operations emphasis graduate, was a collection of approximately 80 questions derived from the FAA knowledge test bank. In general there were few systematic problems noted with the exceptions discussed below.
- Questions from the instrument rating test bank indicated a systematic weakness in the following areas:
- Analyzing weather charts, primarily the surface analysis chart;
- Identifying low altitude enroute chart features, particularly the minimum crossing altitude limitations;
- Static system failures and its detection; and
- IFR fuel requirements.
These areas are being addressed in the current offering of ITEC-291, Instrument Pilot Theory, with a change in the teaching methodology for the course. The current offering is emphasizing real world flight planning exercises in more challenging parts of the United States. It is anticipated that these actual flight planning exercises will increase both understanding and retention of this information.
- The answers to the questions used to assess the graduates’ knowledge of multi-engine operations were universally unacceptable. In one case an individual missed seven out of ten questions. However, further examination indicated that the particular individuals taking the assessment had enrolled in, but not completed the multi-engine rating. Thus, the individuals had signed up for the course which then allowed them to apply for graduation and thus triggered the assessment process but had completed little if any of the required preparation. This is viewed as an administrative, not an academic, problem and steps have been taken to prevent future occurrences by withholding the students’ ability to sign up for ITEC-377 without faculty advisor permission.
ASM - Comprehensive Exam Analysis Suggestions:
The Comprehensive Exam direct assessment of the ASM program is the most troublesome in the opinion of program faculty. In general, program faculty does not believe many of the departmental wide direct assessment questions truly measure against stated competencies. In this line, the ASM faculty has suggested a meeting of the department wide curriculum committee to reevaluate the department wide direct assessment process.
Additionally, the ASM faculty is reevaluating the use of the FAA test question bank as a portion of the direct assessment process for the flight operations emphasis graduate. Currently two trains of thought are being investigated; 1) utilizing the existing FAA Multi-Engine Practical Test as the final direct assessment, or; 2) developing a flight scenario that the graduate would need evaluate and “fly” on the UNK flight simulator. This scenario could involve all aspects of a typical commercial flight operation including emergency procedures.
ASM - Closing Thoughts and Comments:
As previously discussed, the graduates currently participating in the ASM assessment process are actually graduating from the former Airway Science program. Obviously there is some overlap between the programs and in those areas meaningful assessment data has been obtained and utilized. However, it is the author’s opinion that the most significant outcome of this process was to allow the ASM program faculty to further refine the assessment process.
Construction Management (CM) - 2006
In the academic year of 2005-06 the CM program conducted a Graduate Survey, Employer Survey, Student Confidence Scale and a Final Evaluation by Worksite Supervisor Analysis. The results of these instruments were taken to the CM Advisory Committee on October 13, 2006 for discussion. The new direct measures Comprehensive Exam was given for the first time in the ITEC-498 capstone course in the Spring semester of 2006. The data gathered from these different instruments are added to a data bank to determine trends and issues that need to be addressed by the program. The following is a summary taken from the different instruments and the suggestions from the CM Advisory Committee.
CM - Graduate Survey Analysis
Thirty-six surveys were sent out in the Summer of 2006 with 19 respondents. This was a return of 52% which is exactly the same percent of return as the year before. The students rated the general studies courses low but since the ITEC department has no control over the general studies program we have elected not to make any recommendations for changes. This is the responsibility of the General Studies Council that controls the program on the entire UNK campus.
One item that came from the survey is the need for current software and equipment. The CM Advisory Committee looked at the software being used and recommended that we use Timberline, P3, Excel, Revit, and AutoCad. This information will be used in the future when strengthening the courses. A strength of the program is the placement rate of the graduating seniors. From the survey it was determined that 95 percent of the students had positions within two months of graduation.
CM - Employer Survey Analysis
The return rate was not what the CM faculty would have liked with the final percent being 45. One trend that is students are not paying enough attention to detail while on the job sites. The CM Advisory Committee and faculty discussed this issue and decided to include more detail in the classroom in the hopes of stressing the importance of completing all the details of a project. Another trend that has come from the survey is that students today do not have much tolerance for stress. Current students have been raised under a different culture than the students of 30 years ago. This is something the faculty will need to study and address in the future. Written and oral communications could also be improved. One method that is going to be used is not only having students write and give oral presentations but to have them in situations where they are answering questions under fire. Being able to think quickly and under pressure is something that needs to be improved.
CM - Student Confidence Survey Analysis
Twelve instruments were sent out with eight returning for a 67% return rate this year. One of the bright spots was that of safety. The students felt that they had been properly prepared to handle the safety issues on the job site. The CM Advisory Committee was pleased with the results but still stressed that this topic should always be number one. As a result we have changed our safety program so that all students graduating will have a 10 hour Occupational Safety and Health (OSHA) card. This will assist students in getting a construction position and better prepare them for running a safety program. Project planning needs some improvement so the curriculum in the ITEC-498 class is being changed to help with this topic. The CM Advisory Committee suggested that students be given more project management situations in the classroom and have the students better able to answer questions concerning those problems. This suggestion will be implemented into the CM program.
CM - Final Evaluation of Intern by Work Site Supervisor Analysis
The Final Evaluation of Intern by Work Site Supervisor instrument pointed out the same trends as the Employer Survey. The CM Advisory Committee discussed these two measures at the same time. One item that was discussed was how they expect a student to dress when interviewing. It was recommended that students should wear a shirt, tie and sports coat or suit. This information will be made available to the students in the classroom and when doing their internships.
CM - Comprehensive Exam Analysis
The CM Comprehensive Exam, a direct measure, was given to the students in the ITEC-498 capstone course for the first time in the Spring of 2006. It was divided into two sections. The first was the core information which all ITEC students are exposed to and expected to know and the second being questions that are construction management in nature. Since this is the first time this exam has been given the faculty will go through the test to determine reliability and validity. In the Spring of 2007 the exam will be given again and the faculty will begin evaluating the results to determine trends, and from those trends, a plan will be developed and implemented for improvement.
CM - Closing Thoughts
Information on trends and issues are collected from the assessment instruments. When used in conjunction with the CM Advisory Committee, the results do help to improve the CM curriculum and keep it current with the construction industry.
Industrial Distribution (ID) - 2006
During the Spring and Summer of 2006 the Industrial Distribution (ID) program collected and analyzed data from two direct and three indirect assessment measures. The measures include the Graduate Survey 2006 Report, Employer Survey 2006 Report, Student Confidence Scale 2006 Report, the Comprehensive Exam 2006, and the Final Evaluation of Interns by Work Site Supervisor Summary Report from the Spring of 2005 through the Summer of 2006. These measures are used to help determine the degree to which current ID students, past graduates of the program, and their employers feel ID students and graduates are meeting satisfactory levels of performance against the established ID student outcome statements. The following is an analysis of each of the assessment instrument results.
ID - Graduate Survey Analysis
Sixty-one ID alumni were invited to respond to the Graduate Survey. A total of 36 responses were received for a return rate of 59%. This represents a slight increase in respondents from the previous year.
Eighty-eight percent of the ID alumni responding to the survey indicated their primary objective for attending UNK was to prepare for immediate entry into a career and 91% claimed this objective was achieved. Over 97% rated overall satisfaction with the quality of their learning experience as good to excellent and would recommend UNK to others. It speaks highly for the university and the program when almost all ID alumni said they would recommend to others that they attend UNK. Noteworthy is the fact that 50% of alumni plan to seek a Masters degree at some point in their career.
Within the UNK Student Services Categories (eleven listed), students were again most satisfied with career planning and job resources, followed by recreational / athletic / cultural opportunities. This speaks highly for the university as a whole and the ID program in particular which holds two careers fairs each year specifically devoted to ID majors.
As was the case last year alumni were least satisfied with UNK financial aid services followed by tutorial services and student organizations. The ID program is actively – and successfully – acquiring new sources of funds for scholarships to help increase the satisfaction level. Scholarships specifically issued from the ID program now account for 59% of all scholarships given out by the ITEC department and 23% of all scholarships awarded by the College of Business and Technology. The ID faculty continues to make progress in this vital area.
The ID program is also making important changes to the structure of the Industrial Distribution Organization (IDO), the student run organization for ID majors. Starting in the Spring semester of 2007, the IDO members will be using a point system or a "Levels of Achievement" format where the members will have many choices or opportunities to provide service acts for the organization. Service acts include activities such as community involvement, recruiting new majors to the program, going to high schools to promote the ID program, assisting with setup at meetings, career fairs, and other ID functions. All companies associated with the ID program will be made aware of these activities. Students who have earned the rank of bronze, silver, gold, and platinum will be recognized at an ID awards night banquet. The students will also receive recognition on the ID website.
From the Academic Service Categories (nine listed), students were most satisfied with faculty concern for students, courses in the major, and faculty availability. This data represents the same findings reported the previous year and the faculty is pleased with the results. Again, these findings speak highly for the faculty. As was the case last year, ID students were least satisfied with the general studies coursework. Student comments indicate they did not have an understanding of how general studies coursework is, or would become, valuable to them in the workplace. The department has faculty representation on the General Studies Committee responsible for the program campus wide and will continue to address this issue with the committee.
From the Facilities and Equipment Category (six listed) the majority of alumni were satisfied with the facilities and equipment in the ITEC department, especially the availability of computer resources for general and major coursework. There was some slight dissatisfaction with facilities and equipment noted by a few alumni but not to the degree it represents a concern for the faculty.
When it comes to employment, 100% of ID graduates started work within 6 months of graduation with the majority (77%) being employed immediately into their job following graduation. A full 94% indicated they were employed in a position directly related to their major course of study. Ninety-eight percent indicated UNK did an adequate to excellent job of preparing them for their first position after graduation.
The value to companies offering an internship is also clearly apparent with 47% of graduates currently working for the company where they completed their internships. Eighty-nine percent also indicated they have received increased responsibility since starting work with 21% now supervising others and 68% working with minimal supervision.
Industrial Distribution graduates continue to be flexible when it comes to relocation with alumni now working in 16 states. Graduates are living and working on the West Coast, Pacific Northwest, Rocky Mountain States, Southwest, Midwest, Southeast, and New England. Fifty-four percent have taken employment in the Midwestern states of Nebraska, Illinois, Kansas, Missouri, Ohio, and South Dakota.
With respect to salary, the majority of ID alumni started their first job between $35,000 and $45,000. The overall starting salary range was less than $30,000 (3%) to over $60,000 (9%). Current salaries for ID alumni range from below $25,000 (3%) to over $100,000 (9%). The majority of ID alumni currently earn between $45,000 and $70,000 (54%) and work on average a 40 to 60 hour week.
ID- Employer Survey Analysis
Eighteen employers were invited to respond to the Employer Survey. A total of nine responses were received for a return rate of 56%, a 6% increase from the previous year.
In the Productivity Category (seven listed) employers replied that most UNK graduates were doing an outstanding job in terms of volume, quality, and consistency of work. Knowledge of work, attention to detail, and organizing efficiently received the lowest ratings although they still were rated average to good.
In the evaluation of Business Techniques Category (seven listed) employers rated ID alumni high in all categories (meeting people, working harmoniously with others, telephone techniques, following instructions, accepting criticism, and in oral communication). If there was areas to be sensitive to, meeting people and working harmoniously with others may be areas for further study. Data collected over the next several years should determine if any changes to the program should be made to strengthen these skill areas.
In the Work Performance Characteristics Category (12 listed), UNK alumni were ranked highest by employers in initiative, accuracy, and appearance and dress. Somewhat lower scores, while not unacceptable, were noted for judgment, patience, and flexibility. These categories will be watched in subsequent years for trends to indicate if attention is required to strengthen these areas.
Overall, ID employers continue to be pleased with the caliber of UNK graduates, with 83% indicating they were pleased (an average of 4.33 on a 5 point scale). The faculty, while satisfied with these results, will strive for even further improvement.
ID - Student Confidence Scale Analysis
Nineteen students were invited to respond to the Student Confidence Scale Survey for 2006. A total of 9 responses were received for a return rate of 47%. This is markedly lower from the previous year of a 94% return rate. The reason for this has not been established and is under review. It does however point to an apparent problem and the ID faculty needs to take an active role in identifying the root cause and resolving the issue.
Overall, students responding to the Student Confidence Scale reported above average ratings on their confidence level to be proficient in accomplishing each of the 70 different tasks (student outcome statements) presented on the instrument. A seven point scale was used with 1 representing insufficient ability and 7 representing excellent ability.
The lowest ability score (achievable goals) was rated 4.57; the highest (explain career path options in the field) was rated at 6.71. Sixty-one of the 70 tasks were rated 5.5 or higher. Overall, the faculty is very pleased with the outcome of these findings and is addressing the causes of the lower scores.
ID - Final Evaluation of Intern by Work Site Supervisor Analysis
Forty-four people were invited to respond to the Final Evaluation of Intern by Work Site Supervisor Survey. A total of 43 responses were received for a return rate of 98%. The Final Evaluation of Intern by Work Site Supervisor is a summary of individual comments and evaluation of work performance in terms of productivity, business techniques, and personal behaviors.
In the Productivity Category (seven listed) interns were rated highest in interest in work and volume and quality of work performed. While still rated good to outstanding, interns were ranked lowest in organizing efficiently. This compares favorably with data received in the previous year.
In the Business Techniques Category (10 listed), interns ranked highest in their ability to work harmoniously with others and in meeting people. While most scores were good to outstanding, they rated lowest, although not unacceptable, in telephone techniques and accepting criticism. This is consistent with the finding in the 2005 survey.
In the category of Personal Behaviors Category (16 listed), the interns were ranked highest in dependability, flexibility, cooperation, willingness to work, initiative, and accuracy. While most scores were good to outstanding, they rated lowest in leadership followed by judgment, creativity, and self-confidence.
Only two respondents said they would not be interested in having another intern from our program. The majority of respondents are very much interested in hiring more interns from UNK.
Given the consistency of data from year to year the faculty will consider changes to the program to further strengthen “apparent” weaknesses as reported in this measure.
ID - Comprehensive Exam Analysis
The Comprehensive Exam for ID was administered for the first time in the Spring of 2006. Since this was the first time the instrument was administered little conclusive interpretation of the data can be made. The data from this instrument will be evaluated in more detail over the next several years.
However, it is clear from the first administration of the exam that there are questions that are somewhat ambiguous and are in need of refinement. This will be accomplished before the Spring of 2007 when the exam will be given for the second time. It was noted during faculty discussions that questions asking for specific details and precise calculations may not be the best approach to measure overall learning. It is planned to revise a number of questions to be more conceptual in nature.
ID - Conclusions, Recommendations
On November 10, 2006 the latest ID survey data along with faculty recommendations were sent to the ID Advisory Committee for review and preparation for discussion at the ID Advisory Committee meeting held November 14, 2007. All advisory committee members had read the recommendations provided prior to the meeting. During the meeting discussions took place regarding faculty recommendations and all were agreed to without any additional recommendations being presented.
After careful review of the results from all five assessment measures, the ID faculty and the ID Advisory Committee, believes that weaknesses mostly appear to be in the areas of “finance and accounting,” and in “organizing work efficiently.” Specific changes to the program are being, or have already been, implemented to address these areas of weakness.
In the ITEC-452 course, Branch Operations Management, a unit on Distributor Finance was added and has five new modules: (1) How Distributors Make a Profit, (2) Financial Fundamentals, (3) Play the Percentages, (4) Set & Protect Price, and (5) Optimizing the Five P’s.
In the ITEC-271 course, Industrial Products and Applications, two selling units have been added to better cover “Selling Skills” and “Account and Territory Management.” Although these topics are covered again in other forms in upper level classes, this strategy will strengthen student knowledge and organizational skills.
Selling Skills has 3 new modules: (1) Pre-Call Planning and Opening the Call, (2) Identifying and Targeting Customer Needs, and (3) Overcoming Objections and Closing the Sale. Account and Territory Management has 3 new modules: (1) Time Management, (2) Managing Accounts for Growth and Profitability, and (3) Managing Your Territory for Maximum Productivity.
Telecommunications Management (TM) - 2006
The Telecommunications Management program, for the 2006 Academic Year, again conducted five assessment studies: Graduate Survey, Employer Survey, Student Confidence Scale, Final Evaluation of Interns by Work Site Supervisors, and the Comprehensive Exam. These reports were shared with the TM Advisory Committee for their evaluation via the internet and in preparation for the December 7, 2006 TM Advisory Committee. The committee met with the faculty to review and discuss the survey data and faculty recommendations. The following represents the discussion and acceptance of survey data and recommendations as presented to TM faculty.
TM - Graduate Survey Analysis
Twenty-five graduates were invited to complete the Graduate Survey, with seventeen surveys completed resulting in a response rate of 72%. The survey indicated that 82.35% of the graduates had attained employment either prior to graduation or immediately following. All but one of the graduates’ first positions were either directly related or somewhat related to TM and they all agreed that their preparation at UNK was adequate to excellent for the position. The survey indicated that 100% of the students would recommend UNK to their friends.
Graduate’s again this year noted the need for a background in network server setup and security. The new offering by the Computer Science and Information Systems (CSIS) department in the area of Server Area Networks (SANs) will address part of this request. As noted previously, development work is continuing on a network security course in the CSIS department. The TM Advisory Committee supported the need for course work in the areas of SANs and network security.
The TM Advisory Committee was impressed with the Graduate Survey results. The majority of students this year were either satisfied or very satisfied with university registration, career planning and job resources, and academic advising. Career planning and job resources are managed by three specific areas which are continuously communicating with each other: UNK Career Services, the Industrial Technology Internship Director, and the faculty of the TM program. The TM Advisory Committee feels that the student required advisor visit, prior to allowing the student access to the on-line registration procedure, is working well for both advising and career planning.
TM - Employer Survey Analysis
Nine employers were invited to respond to the Employer Survey for 2006. Five completed responses were received for a 66% response rate. Employers noted that program graduates had been hired for positions with the following titles: IT Manager, PC Technician, Management Trainee, Plant Assistant and Broadband Technician. Employers stated that 80% of the graduates had been given increasing responsibility. All of the graduates had earned performance evaluations for productivity at or above a rating of average with 80% of those rated good or outstanding. All graduates were rated by their employers on work performance for business techniques and performance for personal characteristics at or above average with 80% of those rated good or outstanding. Employers generally praised the program and were satisfied with the students’ knowledge and ability to work with people.
TM - Student Confidence Scale Analysis
Seven students were invited to respond to the Student Confidence Scale. Four of the seven surveys were returned for a completion rate of 71%. Students were asked to evaluate their ability to complete 49 different tasks. A 7 point scale was used with 1 representing insufficient ability and 7 representing excellent ability. The lowest average ability score was 5 with the highest average score being 7.
Task number forty-nine received the lowest average score and dealt with “Understand macro and micro economic fundamentals.” In an effort to address this issue the TM Advisory Committee is evaluating curricular changes.
Task number one received the second lowest average score of 5.25 and addressed the students’ ability to “explain complex/technical ideas & concepts to non-technical persons both verbally and in writing.” This is the second year for this task to be rated as the second lowest of the 49 tasks. This issue is being addressed by increasing the number of discussions and the number of presentations given throughout the TM program dealing with technical interpretation. Additional activities have been added to the Threaded Case Study, which spans the Cisco Networking Academy (CCNA) four course sequence, to have the students both present verbally and in writing difficult and technical issues in layperson terms. The results of these additional measures will be evaluated by the Telecommunications Faculty and by the Telecommunications Advisory Board at the annual May 2007 Board Meeting.
TM - Final Evaluation of Intern by Work Site Supervisor Analysis
The two areas which employers last year noted would generally improve the student’s potential are the ability to implement servers and a background in network security. The CSIS is presently offering a course addressing the establishment of Server Area Networks (SANs) for the benefit of both the TM and CSIS students. The CSIS department is also developing a course in Network Security which will meet both departments’ needs.
This year, one employer noted the need for Linux server experience and also noted that additional work with FTP, SSH Client would be beneficial. This content is being addressed in the new CSIS server course. Two employers recommended additional experience with subnetting. Additional emphasis will be placed on this content in ITEC-335 and ITEC-345.
A percentage of 70% of the employers were very interested in hiring another intern. Over half of all employers surveyed would be interested in serving on the TM Advisory Committee. At least 70% of the employers surveyed felt that the student’s work performance in the areas of productivity, good business techniques, and personal ability rated at the level of good or outstanding.
TM - Comprehensive Exam – Analysis
This was the first year for the TM program to administer the Comprehensive Exam. The following information and advisory committee responses were obtained.
Eighty questions were selected to represent a comprehensive exam utilized to directly evaluate student comprehension in the TM program. A sample of eight students majoring in TM and four students completing a minor in the field responded to the exam. Three questions answered by the majors received no correct answers. The questions dealt with imaginary space relationships, how VLANs are made visible to each other and the last dealing with National Fire Protection Association label symbols.
The TM Advisory Committee felt that the content which these questions represented should be further emphasized. Ten of the questions received only one correct response from the eight majors and were related to the areas of drafting, management, electricity, electronics, TCP/IP protocols, communications law and ISDN. The advisory committee reviewed the exam results and felt that these issues could be addressed within the present curriculum frame work and did not require the addition of new classes.
The sample size for the minors was four students and thus some information was certainly gained. However, reliability is limited. Six questions answered by the students seeking a minor in TM received no correct answers. These were in the areas of recognizing planes intersecting a right circular cone, drafting tolerance symbols, supervisors dealing with productivity issues, electronics, cut-through packet switching technology and methods of amplifying signals within the fiber.
The advisory committee noted that the TM minor is not required to take course work in the management and electronics areas. They also felt, as with the major, that additional emphasis could be placed on these areas within the present curriculum and there was no additional courses needed at this time.
During the Spring semester of 2006 the faculty of the Department of Industrial Technology again considered the following points:
- Are the desired outcomes for graduates of the department relevant and defensible?
Upon reviewing student outcomes and the measures used to assess the outcomes, the faculty of the department continues to believe the answer to the above question is yes. It is also important to note that the program advisory committees are an integral part of the review process and continue to provide the ITEC department with valuable input.
2. Do the current means of assessing actually assess the department’s desired outcomes for graduates and provide information that allows for continuous improvement of departmental programs?
Given that the process allows for assessment to occur at both the course level, at the end of the internship, and after graduation, the faculty believes that the data collected will allow for program improvement.
3. Is the scope and focus of our assessment process reasonable?
The faculty believes that while the process is time consuming, it continues to be increasingly more reliable. Over the years that the department has been collecting data, the survey instruments have changed in scope and size as the assessment process has evolved. The data that has been collected has prompted the faculty to review and update the curriculum and internship experience. Many significant changes have been implemented throughout the programs as noted in the analysis discussions.
4. Do we need to discontinue or add any assessment activities?
The faculty constantly evaluates its existing curriculum, internship program, and assessment activities. At this point the faculty believes that the assessment practices are strong and provide excellent feedback to the program. The faculty feels that the assessment activities clearly meet accreditation standards.