Assuring Validity Of Summative Outcomes Evaluation
The terminal goal is to assist in assuring the validity of the outcome evaluation upon which the Florida A&M University, NSF HBCU-UP project will base its summative conclusions regarding the overall success or failure of its initiatives and efforts.
Objective
A. Advise & Collaborate In Evaluation Design
A. Collaborate with Florida A&M University's designated assessment team to provide collaborative advice to the project's leadership on planning and implementation of appropriate and valid evaluation design, measurement & assessment processes.
Indicator
A. Phone Conferences & Email On Evaluation Design
A. Evidence of phone meetings and email exchanges with leaders of the FAMU HBCU-UP to be briefed on evaluation plans developed by the group at various points in the project's unfolding and to brainstorm about the project's evaluation design.
Criterion
A. One Phone Meeting And Two Email /document Exchanges On Evaluation Design
A. One phone meeting and two email /document exchanges per semester addressing evaluation design: either initializing, revising, or confirming and checking up on the project's planned evaluation design and specific elements therein.
Finding
Remote Exchanges Updating Implementation And Evaluation Design
o During the course of the Fall semester there were two phone calls and email exchanges from consultant requesting the updating of implementation and evaluation design and requesting assessment. the client did not provide either details of program changes and or data for examination by the consultant
Indicator
A. Face To Face Meetings On Evaluation Design
A. Document and count phone meetings with leaders of the FAMU HBCU-UP for briefings on evaluation plans developed by the group at various points in the project's unfolding and to brainstorm about the project's evaluation design.
Criterion
A. One Face To Face Meeting
A. Meet face to face at least once during the contract with FAMU project leadership, assessment personnel and persons who will implement program strategies, on site, at FAMU, and off-site (to be arranged) to develop necessary project background & understanding . This is important for developing background to help with advising on design, for understanding assessment implementation challenges and practical solutions, and for interpreting assessment results
Finding
Spring Semester Face To Face Meeting In Huntsville And Orlando
o In April of 2009 Consultant met with FAMU client in Huntsville to recap project implementation design changes and to discuss appropriate corresponding changes in the evaluation design. A second Face to face meeting occurred in Orlando during the project's Advisory Board meeting, at which project implementation changes were again detailed and the consultant presented the revised evaluation design and plan. No data was supplied to the consultant by the client for review.
Indicator
A. Client Evaluation Of Consultant On Advice In Evaluation Design
A. Client evaluation of Consultant collaboration and advice on assessment and evaluation design
Criterion
A. Client Evaluation On Evaluation Design
A. Favorable client evaluation with regard to helpfulness of Consultant's advising on evaluation design
Finding
Client Evaluation Re Evaluation Design
o Evaluation activity within the project has been minimal and the evaluation design has not yet been a major consideration in driving planned assessment due to the fact that attempting to implement the project design has been a major challenge for the HBCU UP program team. Following the face to face meeting between the program director and the Evaluation consultant in Huntsville and the communication of an update on program implementation changes there was considerable favorable informal feedback from the client team about the consultant's evaluation design modification to accommodate the program implementation changes. The first oppty for the FAMU team to evaluate the suitability of the consultant's evaluation design in application will come in Fall of 2009
Action
Evaluation Design Integrated With Program Design
Following communications leading up to the face to face meeting between the program director and the Evaluation consultant in Huntsville the consultant restructured the presentation of the evaluation design by integrating it within a quasi experimental program design for the entire project. In response to the information communicated during the meeting, the program and evaluation design was further refined. It was presented at the April Advisory Board Meeting in Orlando FL and was well received by program leaders and advisors.
During the Tele-conference scheduled for late August of 2009 with the key project FAMU HBCU Up personnel and members of other offices and programs that support aspects of the project on the FAMU campus, the groundwork will be laid with the assessment worker-bees at FAMU for retroactively assembling formative assessment results that will permit the program leaders and the consultant to determine which targeted project performance, learning, and student development outcomes have been achieved. We will then lay the groundwork for scheduling the assessments that will produce the 09-10 formative evaluation results. However, some modifications may be required in the 09-10 assessment implementation based upon the retroactively organized and analyzed formative results for the 08-09 and the 07-08 years of the project's life.
Objective
B. Assessment Implementation Input And Review
Provide input on challenges to planned assessment implementation. Also, provide recommendations for improvement of student-outcomes assessment implementation
Indicator
B. Evidence Of Assmt Review Feedback
B. Documentation and other evidence of informal feedback given to client in response to review of planned assessment implementation
Criterion
B. One Written Assessment Implementation Review
B. Assuming that the client has provided the client with a written information about planned assessment implementation and copies ,or written descriptions of measurement instruments, or assessment tools to be used, at least one written report or reaction to the implementation plan and materials must be provided to the client each year.
Indicator
B. Evidence Of Response To Need For Methodological Assistance
B. Documentation and other evidence of having responded to FAMU client requests for input on methodological challenges that arose in implementing assessment
Criterion
B. One Instance Of Methodological Assistance
B. Assuming that the client requested assistance with a methodological challenge or that Consultant noticed such a challenge and responded to it without being asked there should be evidence of technical methodological assistance having been provided
Finding
Evidence Of Response To Need For Methodological Assistance
The client was not yet in a position to call for assistance with a methodological challenge and the Consultant did not have the oppty to examine assessment proceses and products in order to note any challenges requiring technical assistance.
Indicator
B. Client Evaluation On Assmt Implementation Input
B. Client evaluation of Consultant input on Assmt Implementation
Criterion
B. Client Evaluation On Assessment Implementation
B. Favorable client evaluation with regard to helpfulness in advising on assessment implementation
Finding
Client Evaluation On Assessment Implementation
Client provided favorable evaluation re. helpfulness of consultant's advice on evaluation implementation. Unfortunately, client was unable to produce artifacts and evidence of assessment implementation for consultant review. As a result no concrete advice update could be delivered by the consultant. Hence this assessment was not conducted. This assessment will be delayed until the fall of 2009, after client has had the oppty. to present assessment for the consultant to examine. See actions for more info.
Action
Help With Launch Of Formative Assessment To Be Provided In August 2009 Meeting
During the Tele-conference scheduled for late August of 2009 with the key project FAMU HBCU Up personnel and members of other offices and programs that support aspects of the project on the FAMU campus, the groundwork will be laid with the assessment worker-bees at FAMU for retroactively assembling formative assessment results that will permit the program leaders and the consultant to determine which targeted project performance, learning, and student development outcomes have been achieved. We will then lay the groundwork for scheduling the assessments that will produce the 09-10 formative evaluation results. However, some modifications may be required in the 09-10 assessment implementation based upon the retroactively organized and analyzed formative results for the 08-09 and the 07-08 years of the project's life. .
Objective
C.Provide Summative Documentation Of Improvement Recommendations Or Confirmation Of Success
As part of the summative evaluation, provide documented feedback on implementation methods and procedures which could have been improved. Also Provide documented confirmation of successful methods and results.
Indicator
C. Evidence Of Feedback On Validity Of Results
C. Written documentation of feedback on assessment results vis a vis technical validity, reliability and authenticity
Criterion
C. One Feedback Document On Formative Results And Methods
C. At least one written document of feedback, recommendations or confirmation should be deliverd to client on formative results and their associated assessment methods each year of the project and contract duration. This assumes that the FAMU HBCU-UP Project client produces and provides formative assessment results and information on the assessment methods used to obtain them
Finding
Client Evaluation On Sumative Results Feedback
Summative evaluation is not scheduled until the final year of the project. That is still in the future.
Criterion
C. Documentation Of Feedback On Summative Results And Methods
C. Over the course of the contract, at least one written document of feedback, recommendations or confirmation must be delivered to client on summative results and their associated assessment methods. This assumes that the FAMU HBCU-UP Project client produces and provides summative assessment results and information on the assessment methods used to obtain them
Finding
Documentation Of Feedback On Summative Results And Methods
Summative results will not be available this year, hence feedback was not provided
Indicator
C. Client Evaluation On Written Feedback Of Formative Results And Methods
C. Client evaluation of Consultant's written feedback on formative results and methods
Criterion
C. Client Evaluation On Formative Results
C. Favorable client evaluation with regard to helpfulness of written feedback on formative results
Finding
Client Evaluation On Formative Results
Premature. Summative evaluation results are not expected for another two years.
Indicator
C. Client Evaluation On Written Feedback Of Summative Results And Methods
C. Client evaluation of Consultant's written feedback on summative results and methods
Criterion
C. Client Evaluation On Sumative Results Feedback
C. Favorable client evaluation with regard to helpfulness of written feedback on sumative results
Finding
Client Evaluation On Conultant Feedback On Sumative Results
Premature. Summative evaluation results are not expected for another two years.
Action
Summative Evaluation Input And Feedback Planned For Last Yr Of Program
No Action was planned for this Objective until the final year of the project.
Goal
Validity Of Formative Outcomes Assessments
The intermediate goal is to assist in assuring the validity of interim student outcome measures and of the formative strategy assessments upon which the Florida A&M University, NSF HBCU-UP project will base decisions about program development and improvement.
Objective
A. Advise & Collaborate In Evaluation Design
A. Collaborate with Florida A&M University's designated assessment team to provide collaborative advice to the project's leadership on planning and implementation of appropriate and valid evaluation design, measurement & assessment processes.
Indicator
A. Phone Conferences & Email On Evaluation Design
A. Evidence of phone meetings and email exchanges with leaders of the FAMU HBCU-UP to be briefed on evaluation plans developed by the group at various points in the project's unfolding and to brainstorm about the project's evaluation design.
Criterion
A. One Phone Meeting And Two Email /document Exchanges On Evaluation Design
A. One phone meeting and two email /document exchanges per semester addressing evaluation design: either initializing, revising, or confirming and checking up on the project's planned evaluation design and specific elements therein.
Finding
Remote Exchanges Updating Implementation And Evaluation Design
o During the course of the Fall semester there were two phone calls and email exchanges from consultant requesting the updating of implementation and evaluation design and requesting assessment. the client did not provide either details of program changes and or data for examination by the consultant
Indicator
A. Face To Face Meetings On Evaluation Design
A. Document and count phone meetings with leaders of the FAMU HBCU-UP for briefings on evaluation plans developed by the group at various points in the project's unfolding and to brainstorm about the project's evaluation design.
Criterion
A. One Face To Face Meeting
A. Meet face to face at least once during the contract with FAMU project leadership, assessment personnel and persons who will implement program strategies, on site, at FAMU, and off-site (to be arranged) to develop necessary project background & understanding . This is important for developing background to help with advising on design, for understanding assessment implementation challenges and practical solutions, and for interpreting assessment results
Finding
Spring Semester Face To Face Meeting In Huntsville And Orlando
o In April of 2009 Consultant met with FAMU client in Huntsville to recap project implementation design changes and to discuss appropriate corresponding changes in the evaluation design. A second Face to face meeting occurred in Orlando during the project's Advisory Board meeting, at which project implementation changes were again detailed and the consultant presented the revised evaluation design and plan. No data was supplied to the consultant by the client for review.
Indicator
A. Client Evaluation Of Consultant On Advice In Evaluation Design
A. Client evaluation of Consultant collaboration and advice on assessment and evaluation design
Criterion
A. Client Evaluation On Evaluation Design
A. Favorable client evaluation with regard to helpfulness of Consultant's advising on evaluation design
Finding
Client Evaluation Re Evaluation Design
o Evaluation activity within the project has been minimal and the evaluation design has not yet been a major consideration in driving planned assessment due to the fact that attempting to implement the project design has been a major challenge for the HBCU UP program team. Following the face to face meeting between the program director and the Evaluation consultant in Huntsville and the communication of an update on program implementation changes there was considerable favorable informal feedback from the client team about the consultant's evaluation design modification to accommodate the program implementation changes. The first oppty for the FAMU team to evaluate the suitability of the consultant's evaluation design in application will come in Fall of 2009
Action
Evaluation Design Integrated With Program Design
Following communications leading up to the face to face meeting between the program director and the Evaluation consultant in Huntsville the consultant restructured the presentation of the evaluation design by integrating it within a quasi experimental program design for the entire project. In response to the information communicated during the meeting, the program and evaluation design was further refined. It was presented at the April Advisory Board Meeting in Orlando FL and was well received by program leaders and advisors.
During the Tele-conference scheduled for late August of 2009 with the key project FAMU HBCU Up personnel and members of other offices and programs that support aspects of the project on the FAMU campus, the groundwork will be laid with the assessment worker-bees at FAMU for retroactively assembling formative assessment results that will permit the program leaders and the consultant to determine which targeted project performance, learning, and student development outcomes have been achieved. We will then lay the groundwork for scheduling the assessments that will produce the 09-10 formative evaluation results. However, some modifications may be required in the 09-10 assessment implementation based upon the retroactively organized and analyzed formative results for the 08-09 and the 07-08 years of the project's life.
Objective
B. Assessment Implementation Input And Review
Provide input on challenges to planned assessment implementation. Also, provide recommendations for improvement of student-outcomes assessment implementation
Indicator
B. Evidence Of Assmt Review Feedback
B. Documentation and other evidence of informal feedback given to client in response to review of planned assessment implementation
Criterion
B. One Written Assessment Implementation Review
B. Assuming that the client has provided the client with a written information about planned assessment implementation and copies ,or written descriptions of measurement instruments, or assessment tools to be used, at least one written report or reaction to the implementation plan and materials must be provided to the client each year.
Indicator
B. Evidence Of Response To Need For Methodological Assistance
B. Documentation and other evidence of having responded to FAMU client requests for input on methodological challenges that arose in implementing assessment
Criterion
B. One Instance Of Methodological Assistance
B. Assuming that the client requested assistance with a methodological challenge or that Consultant noticed such a challenge and responded to it without being asked there should be evidence of technical methodological assistance having been provided
Finding
Evidence Of Response To Need For Methodological Assistance
The client was not yet in a position to call for assistance with a methodological challenge and the Consultant did not have the oppty to examine assessment proceses and products in order to note any challenges requiring technical assistance.
Indicator
B. Client Evaluation On Assmt Implementation Input
B. Client evaluation of Consultant input on Assmt Implementation
Criterion
B. Client Evaluation On Assessment Implementation
B. Favorable client evaluation with regard to helpfulness in advising on assessment implementation
Finding
Client Evaluation On Assessment Implementation
Client provided favorable evaluation re. helpfulness of consultant's advice on evaluation implementation. Unfortunately, client was unable to produce artifacts and evidence of assessment implementation for consultant review. As a result no concrete advice update could be delivered by the consultant. Hence this assessment was not conducted. This assessment will be delayed until the fall of 2009, after client has had the oppty. to present assessment for the consultant to examine. See actions for more info.
Action
Help With Launch Of Formative Assessment To Be Provided In August 2009 Meeting
During the Tele-conference scheduled for late August of 2009 with the key project FAMU HBCU Up personnel and members of other offices and programs that support aspects of the project on the FAMU campus, the groundwork will be laid with the assessment worker-bees at FAMU for retroactively assembling formative assessment results that will permit the program leaders and the consultant to determine which targeted project performance, learning, and student development outcomes have been achieved. We will then lay the groundwork for scheduling the assessments that will produce the 09-10 formative evaluation results. However, some modifications may be required in the 09-10 assessment implementation based upon the retroactively organized and analyzed formative results for the 08-09 and the 07-08 years of the project's life. .