Running head: PROGRAM EVALUATION

Evaluation of Training: Indiana State University and Embry-Riddle Aeronautical University's McNair Scholars Programs

Joseph I. Williams

Industrial Technology Education 695

Rationale of Evaluation in HRD

3/31/2003
Evaluations of Training: Indiana State University and Embry-Riddle Aeronautical University's McNair Scholars Programs

Introduction

To ensure the success of President Bush s education initiative No Child Left Behind, high-quality postsecondary educational opportunities must be available to all students. In keeping with this goal, the Federal TRIO Programs provide outreach and support programs to assist low-income, first-generation college students or minority students in progressing through the academic pipeline from middle school to postbaccalaureate programs. This paper describes the Ronald E. McNair Postbaccalaureate Achievement Project and compares and contrasts how organizations at Indiana State University and Emery-Riddle Aeronautical University evaluate the McNair training programs they offer.

The Ronald E. McNair Postbaccalaureate Achievement Program Award

The primary goal of the McNair Scholars Program is to provide experiences that prepare selected undergraduate students for doctoral study. The overriding goal of the McNair programs is to increase the number of underrepresented students who will obtain doctoral degrees and teach and/or do research in higher education institutions.

The McNair Program provides services throughout the year (summer included) to prepare undergraduate participants for graduate study and assisting them to gain admission to qualify for the graduate programs of their choice. The program is funded by the U.S. Department of Education. Only institutions of higher education and combinations of institutions of higher education are eligible to apply for McNair awards (U.S. Department of Education, 2002).

According to the Office of Postsecondary Education (2003) students who qualify for McNair must be enrolled in a degree-granting program at an eligible institution. In all projects, at least two-thirds of the participants must be low-income, potential first-generation college students (Office of Postsecondary Education, 2003). The remaining participants may be from groups that are underrepresented in graduate education. The Office of Postsecondary Education (2003) states:

The Ronald E. McNair Postbaccalaureate Achievement Program awards grants to institutions of higher education for projects designed to prepare participants for doctoral studies through involvement in research and other scholarly activities. McNair participants are from disadvantaged backgrounds and have demonstrated strong academic potential. Institutions work closely with these participants through their undergraduate requirements, encourage their entrance into graduate programs, and track their progress to successful completion of advanced degrees. The goal of McNair is to increase the attainment of the Ph.D. by students from underrepresented segments of society

Services provided by the McNair programs

Ø Research opportunities for participants who have completed their sophomore year of college

Ø Mentoring

Ø Seminars and other scholarly activities designed to prepare students for doctoral studies

Ø Summer internships

Ø Tutoring

Ø Academic counseling

Ø Assistance in obtaining student financial aid

Ø Assistance in securing admission and financial aid for enrollment in graduate programs

The McNair Scholars Program at Embry-Riddle Aeronautical University

          At Embry-Riddle Aeronautical University, what we doand do bestis teach the science, practice, and business of the world of aviation and aerospace.(Embry-Riddle Aeronautical University, 2003).

          The McNair Scholars Program was established at Embry-Riddle Aeronautical University in Prescott, AZ in 1999 (The McNair Scholars Program at the Prescott Campus, 2002). At the McNair website, there were twenty-four McNair scholars in January 2002 participating in the program (The McNair Scholars Program at the Prescott Campus). 

          David Viger, Ed.D. is the Director of the McNair Program at the Prescott campus. Dr. Viger combines three decades of teaching, administrative, and research experience with a passion and commitment to programs that promote diversity. David has taught physics, biology, and chemistry at both high school and college levels. He holds a B.S. and M.S. in Science Education in addition to a doctorate in Higher Education Administration. He has administrative experience at high school and community college levels as well as at Embry-Riddle. His research and work within honors programs and advisement programs, along with his expertise in designing retention programs, bring unique strengths to his position. Dr. Viger has been a part of the Embry-Riddle community for more than twenty years.

          David Brandstein, Program Coordinator of the McNair Scholars Program, combines extensive graduate school, research, faculty, education, and management experience with a life-long commitment to diversity initiatives. With an M.A. in English and doctoral studies in folklore and anthropology, David has taught interdisciplinary courses for more than 30 years at Brooklyn and Bard Colleges, Yale University, New School University, University of Arizona, Northern Arizona University and Embry-Riddle where he's been an Arts and Science faculty member for 6 years. He's also administered varied educational programs for underserved and underrepresented students, and has been a management consultant specializing in communications and diversity training and development programs for corporations and non-profits.

The McNair Scholars Program at Indiana State University

          At ISU, academic and social opportunities are limitless, with more than 125 majors, small classes (80 percent of which are taught by full-time faculty) and nearly 200 clubs and activities to choose from. (Indiana State University, 2002). Indiana State also offers a variety of first-year programs designed to help students make a successful transition to college life, including special advising and orientation programs, first-year seminars and learning communities (Indiana State University).

          The McNair Scholars Program was established at Indiana State University in Terre Haute, IN., in 1999 (McNair Scholars Program at Indiana State University, 2002). At the McNair in January 2002 there were twenty-one McNair scholars participating in the program (McNair Scholars Program at Indiana State University, 2002).

          Tony Brewer has served as the director of the McNair Scholars Program since January of 2001. He has been with the McNair Scholars program since the programs inception in 1999, initially serving as the academic coordinator for the project. Tony earned his Masters of Science and Masters of Public Administration from ISU in 1997 and immediately went on to serve as Assistant Coordinator and Coordinator for the Academic Opportunity Mentoring Program.

          Mrs. Ruth Greenfield is the McNair Program Academic Coordinator at ISU. Ruth Greenfield began working with the TRIO Programs during her undergraduate studies at Paine College in Augusta, Georgia. She served as President of the Special Services Program her sophomore year. Also, she worked as a tutor and counselor for Special Services and Upward Bound. Mrs. Greenfield earned her B.S. in Mathematics and M.S in Engineering from University of Missouri-Columbia.

Training Evaluation Methods

          In order for McNair Programs to determine ways to better serve and prepare students for graduate study, and to provide alternatives for approaches in providing professional development activities, it is necessary to review the methods currently utilized by the program and to determine if program recipients are performing well.

Reaction

          Reaction level evaluations are subjective and may best be defined as how well trainees like a training program. It provides an opportunity for the learners to reflect on the learning experience and to express their level of satisfaction with it. According to Kirkpatrick (1996) it's vital to measure participants' reactions in an organized fashion using written comment sheets that have been designed to find the desired reactions.

          Kirkpatrick noted that comments should also be designed so that they can be tabulated and quantified (1996). Evaluation results at this level alert course developers and instructors to areas of course design and delivery, which may need revision given a consistent number of low ratings. The training coordinator, director, or other trained observer should make his own appraisal of the training in order to supplement participants' reactions so that the combination of two evaluations is more meaningful than either one by itself (Kirkpatrick, 1996).

          At ISU, overwhelmingly scholars report high levels of satisfaction with McNair Program.  This was apparent in both focus groups and in survey data. In fact, McNair scholars are not hesitant to recommend the McNair Program to others. This indicates that on a general level students enjoy whats happening in the McNair Program. The same information from Embry-Riddle was not availabale for research.

Both ISU and Emery-Riddle Universities conduct reaction assessments using printed forms. In addition, Embry-Riddles McNair Program has developed electronic forms available at their website for participant comments (See Appendix 1 and 2).

Learning

          An objective determination of the amount of learning that takes place is important according to Kirkpatrick (1996). Learning has been defined as what principles, facts, and techniques were understood and absorbed by trainees. To help measure learning Kirkpatrick (1996) suggests that facilitators should measure the learning of each trainee so that quantitative results can be determined and use a before-and-after approach so that learning can be related to the program.

          As much as possible, the learning should be measured on an objective basis. In addition, where possible, a control group (not receiving the training) should be used to compare with the experimental group that receives the training and where possible, analyze the evaluation results statistically so that learning can be proven in terms of correlation or level of confidence (Kirkpatrick, 1996).

Behavior

          Evaluation of training in terms of on-the-job behavior is more difficult than reaction and learning evaluations. For evaluating training in terms of behavioral change, according to Kirkpatrick (1996) it requires consideration of many factors such as conducting a systematic appraisal of on-the-job performance on a before-and-after basis, performance appraisals should be made by trainees, trainees' supervisors, subordinates, peers, and others familiar with trainees' on-the-job performance. Kirkpatrick also suggested a statistical analysis should be conducted to compare before-and-after performance and to relate changes to the training as well as a post-training appraisal three months or more after training so that trainees have an opportunity to put into practice what they learned (1996).

Results

          The fourth level measures the impact of training on the effectiveness of the organization. By evaluating the outcome of the services currently being provided to scholar in a McNair program, both strong and weak implementations of the training will be identified which will serve as a guide for the facilitator in determining additional or alternative services to be offered by the program. Kirkpatrick also called this fourth level of evaluation the "results level."

I.       Current Statistics found at the ISU McNair Program

A.     Attainment of a bachelor's degree for students in the program:  20 students have graduated from ISU with their bachelors degrees.

B.     Enrollment in graduate programs for students who have graduated from ISU.

a.      Nine students are currently enrolled in masters or J.D. programs.

b.      One student is currently enrolled in a doctoral program.

C.     McNair scholars who have earned a doctoral degree.

a.      Since this is the fourth year for the program, the director indicates that there has not been enough time for a doctoral graduate.

At the moment the Embry-Riddle information was not available to the research.

References

Donald, Kirkpatrick L. (1996, January). Techniques for evaluating training programs. Retrieved April 7, 2003, from American Society for Training & Development. Web Site: http://www.astd.org/CMS/templates/index.html?template_id=1&articleid=20840

Embry-Riddle Aeronautical University. (2003, March 28). The oldest, largest, & most prestigious aviation & aerospace university in the world. Retrieved March 23, 2003, from http://www.erau.edu/0Universe/01/01abouterau.html

Indiana State University. (2002, December). About ISU. Retrieved March 29, 2003, from http://www.indstate.edu/aboutisu/

Jones, J.E., & Bearley, W. L. (1996). Strategies, tactics and strategies for developing leaders. Amherst, MA.: Organizational Universe Systems.

Kirkpatrick, D.L. (1988). Evaluating training programs: the four levels. San Fransisco, CA.: Berrett Koehler.

Lepsinger, R., & Lucia, A.D. (1997). The art and science of 360-degree feedback.

Office of Postsecondary Education. (2003, January). Ronald E. McNair postbaccalaureate achievement program. Retrieved March 28, 2003, from http://www.ed.gov/offices/OPE/HEP/trio/mcnair.html

The McNair Scholars Program at Indiana State University. (2002, November). Retrieved March 20, 2003, from http://web.indstate.edu/mcnairscholars/index1.html

The McNair Scholars Program at the Prescott Campus. (2002, January). Retrieved March 20, 2003, from http://mcnair.pr.erau.edu/index.html

U.S. Department of Education. (2002). A profile of the Ronald E. McNair postbaccalaureate achievement program: 1999-2000. Washington, D.C.: Retrieved March 28, 2003, from http://www.ed.gov/offices/OPE/HEP/trio/mcnair.html

Appendix 1

McNair Scholars Program at Indiana State University

Please answer the following questions honestly.  Place an X in the appropriate box.  If a question does not apply to you, please leave it blank.

Additional comments, feedback, or suggestions regarding the McNair Project:

Please reflect on the following:

Thanks for your cooperation!

Appendix 2

Emery-Riddle Aeronautical University

Ronald E. McNair Scholars Program

Student-Scholars Monthly Report and Self-Assessment

Name­­­­­­­­­­­­­­­­­­­­­­­_____________________________________________ Month/Y ear­­­­_________________

Please complete this form as thoughtfully, honestly and specifically as possible because it will be useful for our program evaluations and in our on-going efforts to support your success. Keep in mind that the information you provide here will be used only in positive ways, never to harm you, not as "test." There is no need to list activities and accomplishments when they don 't apply to a particular month. Feel free to use the back of the paper if you need more writing room. Please make a copy for yourself before submitting it to the Program Director by the end of each month. Since you will submit a progress report at the end of the semester, you might want to consider this a journal-record of what you've done, where you 're going and how the program can best suit you.

I. I participated in the following activities (please list in chronological order, describe as specifically as possible and include some comments on its value in reaching your goals):

McNair Group Meetings: ____________________________________________________________________________________________________________________________________________________________

McNair Mentor Meetings: ____________________________________________________________________________________________________________________________________________________________

Other McNair Meetings (Director Coordinator etc.):

____________________________________________________________________________________________________________________________________________________________

McNair Experiences at Conferences, etc:

____________________________________________________________________________________________________________________________________________________________

Research Activities:

____________________________________________________________________________________________________________________________________________________________

Graduate School Explorations and Other McNair Activities:

____________________________________________________________________________________________________________________________________________________________

II. Comment on your academic and non-academic activities, progress and concerns.

Academic Achievements and Concerns: ____________________________________________________________________________________________________________________________________________________________

Non-McNair, Non-Academic Activities:

____________________________________________________________________________________________________________________________________________________________

III. Respond to the following open-ended questions that are opportunities for you to evaluate your own progress, to provide feedback on how the program can best serve you, and others.

In terms of the goals of the program, what are the highpoints (achievements, successes, etc) of your month? ____________________________________________________________________________________________________________________________________________________________

What are the low points?

____________________________________________________________________________________________________________________________________________________________

What are your plans for the next month and beyond?

____________________________________________________________________________________________________________________________________________________________

What are some of the things you might do to achieve more success for yourself and for others in the program?

____________________________________________________________________________________________________________________________________________________________

Any additional comments on yourself and on the program? ____________________________________________________________________________________________________________________________________________________________

Signature­­­­­­­­­­­­­­________________________________________ Date _________________________