Joseph Williams

April 5, 2002

I. Proposed Type, Model, and Method of Evaluation and Validation of Selection

A. Proposed type of evaluation: Evaluation of Outcome

1. Pros in selecting this type of evaluation:

a. What the client hopes to gain from this evaluation is a determination of new methods to better serve and prepare students for graduate study, along with new approaches to providing professional development for the students in the program.

b. In order for the client to determine ways to better serve and prepare students for graduate study, and to provide alternatives for approaches in providing professional development activities, it is imperative to evaluate the outcome of the methods being currently utilized by the program and to determine if program recipients are performing well.

c. By evaluating the outcome of the services currently being provided to students in this program, both strong and weak implementations of the program will be identified which will serve as a guide for the client in determining additional or alternative services to be offered by the program.

2. Cons in selecting this type of evaluation:

a. Program staff may have differing opinions of what constitutes a successful outcome.

b. By concentrating evaluation efforts on outcomes could potentially overlook other interesting and perhaps essential portions of the program that needs to be evaluated.

B. Proposed model of evaluation: Improvement-Focused Model

1. Pros in selecting this model:

a. Utilizing this model will enable the evaluator to discover discrepancies between: program objectives and the needs of the program participants, program plans and implementation, expectations of the program participants and services which are actually being provided, and projected and achieved outcomes of the program services currently being utilized.

b. Identifying the above-mentioned discrepancies will provide a blueprint for the client to implement program improvements, which is the clients stated goal for having this evaluation conducted.

2. Cons in selecting this model:

a. Program staff could potentially feel threatened by the identification of the above-mentioned discrepancies.

C. Proposed method of evaluation: Formative

1. Pros in selecting this method of evaluation:

a. The client has stated that he would like to identify ways to improve program services and determine additional or alternative approaches to providing professional development activities to program participants. Therefore, it is appropriate for the evaluator to conduct a formative evaluation in which strengths and weaknesses of currently utilized program services can be identified, and recommendations of additional or alternative services can be made to program staff.

b. Conducting a formative evaluation will enable program staff to identify currently provided services that are effective, and which ones may not be as effective. A formative evaluation will serve as a guide in the process of making on-going improvements to the program and should ultimately enhance the outcomes of the program.

c. Utilizing a formative method of evaluation will better serve the needs of the client, as stated in the initial interview.

2. Cons in selecting this method of evaluation:

a. Conducting a formative evaluation could potentially become more time consuming than conducting a summative evaluation.

II. Program Criteria and Standards & How the Interview Helped Focus the Plan

A. Program Criteria and Standards

1. Criteria that reflects the programs intent

a. The McNair Projects mission statement reveals: provides services throughout the year to prepare students for graduate study The client, Tony Brewer, indicates that he would like to know better ways to serve the students who are preparing for graduate school as McNair Scholars. Since the program already provides services we will be examining current services and programs to see if they align with the program intent.

b. The evaluation will focus on identifying potential new services that will reflect both the program intent and will be feasible for the client.

c. We will endeavor to make observations that will correctly frame the objectives of the program intent with this evaluation.

2. Criteria that the staff can influence

a. During the course of the evaluation the evaluator will be seeking information regarding new services. Current staff that already provides services may have ideas and suggestions for new services. These staff members should be able to provide feedback on the effectiveness of existing services.

b. Current program staff could potentially be offended with the image they are not currently providing good services and the evaluator is here to identify ways to replace what they do. The evaluator will work with current staff to indicate that the goal of the initiative is to identify new services and evaluate existing services to better serve the McNair scholars.

c. This initiative identify measures and criteria that will bridge the efforts of the staff and will also reflect the purpose of the program.

3. Criteria that can be measured reliably

a. Trying to measure information with unreliable methods is not effective. Therefore a goal for this evaluation is for all of the measures to be acquired reliably, i.e. our measurements will be repeatable with the same outcomes.

b. To ensure reliability techniques of observation and evaluation will be employed that have a higher tendency to be reliable. These techniques will include focus groups, surveys and interviews.

c. McNair is a social program with general guidelines; discovering information with very high reliability is a challenge for social programs. Using a multitude of evaluation techniques will help overcome this challenge.

4. Criteria that stakeholders participate in selecting

a. Tony Brewer, the client, was involved from the beginning to identify the major focus for this evaluation. Mr.Brewer is a major stakeholder. Other stakeholders identified include, ISU students who are McNair scholars, potential and future McNair students, staff of McNair, the ISU community in general, and the Department of Education who funds the McNair Project. These additional stakeholders were not consulted prior to writing the evaluation plan.

b. The initiative will interview, observe and survey stakeholders and will be very careful to listen to all their feedback. This information will be shared with the client, Mr. Brewer.

B. How the interview process focused the evaluation plan

1. The McNair Project has a broad mission statement, and the client had a purposeful portion he wanted examined. This approach is a general review of what services are in place and identifies additional effective services that can be provided to McNair scholars.

2. The interview process with Mr. Brewer indicated a strong belief in this program, and the initial appearance that current activities were effective. Yet, he believe that having an outside evaluation would be valuable. Though, he also wanted to ensure that we could give him something useable and effective for the current state of the McNair Project. Concentrating on finding programs that were effective and what were additional programs could be effective seemed ideal.

3. After interviewing Mr. Brewer he provided a multitude of documentation which was very helpful. The McNair Project has a mission statement along with detailed descriptions of the program that will help focus the evaluation instruments clearly with the intent of the program and also identify the meaning of the vague term effective. The section of the Program McNair that will be evaluated will be done on the following criterion: Workshops and Seminars to provide information on such topics as selecting a graduate school, the graduate school application process, preparing a personal statements, graduate financial aid, research skills, presentation skills, technical writing skills, strengthening academic skills, Graduate Record Examination (GRE), and more.

III. Clearly Identified Resources, Proposed Data Gathering Strategies, Justification of Statistical Methods

A. Clearly Identified Resources

1. Stakeholders

a. Program Staff

b. Participants in the McNair Program List of student participants.

c. Past/Potential Participants in the McNair Program.

d. Department of Education Provides funding.

e. Financial Aid Provides lists of qualified Students.

f. ISU Community in General.

g. Department of Graduate Studies Next stop for some McNair scholars.

2. Documentation

a. Website URL: http://web.indstate.edu/mcnairsch/

b. Various Publications and Reports from McNair Office.

c. Newsletters and Pamphlets released to participants and to the ISU community.

B. Data Gathering Strategies

1. Surveys of Current Participants

a. Distribute survey during McNair Monthly Meeting

b. General Topic Areas for Questions

i. Participants' expectations.

ii. Participants satisfaction.

iii. Participation and impact of mentors.

iv. Internship opportunities.

v. Value of research components.

vi. Mission Statement.

vii. Graduate Preparation Process.

2. Current Statistics found by McNair

a. Attainment of a bachelor's degree for students in the program.

b. Enrollment in graduate programs for students who have graduated from ISU.

c. Enrollment in a doctoral degree program for students who have graduated from ISU.

d. McNair scholars who have earned a doctoral degree.

3. Personal/Telephone Interviews to Current and Previous Participants

a. Learning experience from professional development sessions.

b. Mentoring within professional development sessions.

c. Opportunity to work independently.

d. Chance to work with others interested in academic achievement.

e. Opportunity to support personal intellectual curiosity.

f. Opportunity to pursue research interests autonomously.

g. Chance to study in one's discipline outside the classroom.

h. Importance and satisfaction within professional development sessions.

4. Focus Groups of Current Students

a. Focus groups will be randomly selected following November meeting. Students will then meet with program evaluation team to answer similar questions to the survey.

b. Criteria same as for Personal/Telephone Interviews.

C. Justification of statistical methods

1. Presently the evaluator is not capable of determining which statistical measures to employ until review of the instruments by someone with the proper training.

2. The evaluator will employ the use of many descriptive stats and qualitative interview reports to show the results from student/staff personal/telephone interviews and focus groups.

IV. Proposed Resources for the Program Evaluation, Cost-Benefit and Cost-Effectiveness Analysis, and Dissemination

A. Resources for the program evaluation

1. Primary funding for this program evaluation will be the expense of the evaluator. McNair project will tentatively provide photocopy needs for surveys.

2. Focus group incentives for students will not be addressed since these students are mandated to attend these meetings.

B. Cost-benefit and cost-effectiveness analysis

1. There is the cost of the evaluators time and effort toward this project. The cost- benefit that comes from the evaluators time and effort in evaluating the services will better prepare students for graduate study as McNair scholars and will help improve the skills of the program evaluator.

2. The outcome of the program evaluation will look at the services being offered by the McNair Project and evaluate their effectiveness in order to give the client, Tony Brewer, information that will allow him to provide better ways to serve the students who are preparing for graduate school as McNair Scholars.

C. Dissemination

1. The report will be given to Mr. Brewer and, with permission of Mr. Brewer, any stakeholder who wishes to receive the final program evaluation report. At the conclusion of the evaluation, the evaluator plans on giving Mr. Brewer the statistical analysis, presentation materials, and any other materials used in the evaluation process.

V. Challenges and Potential violations of Program Evaluation Standards

A. Anticipated challenges to the successful completion of this evaluation and the method(s) chosen to overcome them:

1. Time restrictionsThe short time frame as well as limits imposed by schedule restrictions of the full-time student as evaluator will be compensated for by narrowing focus with input from program director with emphasis to be placed on area deemed most important to Mr. Brewer.

2. Potential apathy of program participantsAlthough unlikely, this must be addressed as a possibility. Participants, also full-time students, with highly demanding schedules will be assured that their participation is of vital importance to the McNair Project and their chance to give back to the program.

3. Follow-up phone interviewsDue to limited time, attempts to reach participants will be restricted to a few occasions.

4. ConfidentialityWhile there is little information to be gathered that might fall under serious confidentiality concerns, each student will be assured that all information will be treated as confidential. Since there is no need to match responses with pre and post test arrangements, tracking responses by an individual is unnecessary.

5. Limited experience of evaluation staffAs a student practitioner, there is little to be done apart from paying strict attention to program evaluation guidelines. The neophyte standing of the evaluator is known to the major stakeholder, Mr. Brewer.

6. Concerns of program stakeholders for the survival of the programIt is vital to present the evaluator as allies in the improvement of the program as it is directly linked to the livelihood of both staff and participants alike.

B. Potential violations of Program Evaluation standards

1. Stakeholder identificationTime limits require abbreviated contacts meaning some stakeholders might be overlooked.

2. Evaluator credibilityThe use of a student evaluator with limited experience requires a tightened focus, and so, a possible omission of important factors in evaluating this program.

3. Political viabilityThere will be limited time for the evaluator to establish rapport with and understanding for the clients served by this program.

4. While other threats may exist, they are primarily a function of time and experience limitations. These have been well documented and made known to the stakeholders with the understanding that they will work with the student evaluator to guard against impinging on the integrity of the evaluation.

References

Donald, Kirkpatrick L. (1996, January). Techniques for evaluating training programs. Retrieved April 7, 2003, from American Society for Training & Development. Web Site: http://www.astd.org/CMS/templates/index.html?template_id=1&articleid=20840

Kirkpatrick, D.L. (1988). Evaluating training programs: the four levels. San Fransisco, CA.: Berrett Koehler.

Lepsinger, R., & Lucia, A.D. (1997). The art and science of 360-degree feedback.

Apendix 1:

Initial Interview with Tony Brewer, Program Director, McNair Project

What is the purpose of the McNair Project on this campus?

To prepare first generation, low-income students for graduate study (PhD).

What do you hope students will gain from the McNair Project?

Gain the positive attitude, knowledge and the perception that they can successfully become PhD students.

What are the short-term goals of the project?

Students maintain at least a 3.5 cum GPA and graduate with a bachelors degree.

What do you see as the major benefit to the community from participating in the program?

Students participate in community service activities, including serving as mentors. The program better prepares low-income students to be positive role models.

Do you have access to lists of ISU students who currently identify themselves with qualifications that would make them a potential candidate for the program? If so, what is done to recruit these students?

Program staff is provided with a list of qualified students from the ISU Office of Financial Aid. Faculty mentors also recommend students for the program. Biggest recruitment method is word of mouth.

What do you see as the strengths of the project at ISU?

Offering opportunities to prepare students well for graduate school and professional development.

What do you see as the areas of challenge (weaknesses) of the project at ISU?

Finding quality students who can appreciate the long-term advantage to pursuing a PhD.

When a client is meeting with a project representative in an appointment, what kind of questions are asked?

What area of study does the student want to pursue is their PhD work? What is their motivation to do so?

How does the project follow-up with clients while an undergraduate at ISU?

Updates on the students academic progress, grades, provide tutoring, help them find employment on campus, attend mandatory monthly seminars, research lab use, and meet once a month with a graduate student mentor.

Where do you see the program in five years? In ten years?

To have at least 10 individuals acquire their PhD. Increase in numbers of students to 25 per year.

Does each student receive the same amount of funding each year?

Yes, $2,800.

Could you identify other stakeholders we could talk to?

Ruth Greenfield, Asst. Coordinator; Tea Uttley, Secretary; Will fax a list of student participants and their phone numbers so we can talk with them as well.

What is it you hope to gain from this evaluation?

In what ways we could better serve and prepare students for their graduate study. Interested in having outside observers provide feedback of existing professional development opportunities. Also providing suggestions of new ways to provide professional development.