Promote An Environment That Encourages Continuous Improvement Of Assessment Initiatives
The Office of Academic Planning and Assessment will encourage and promote an environment of continuous improvement for all departments, offices, and programs within the various Colleges and Divisions at Sam Houston State University.
Objective
Ensure Quality Annual Assessment Processes
The Office of Academic Planning and Assessment will ensure that members of the university community are conducting a quality, and effective annual assessment process.
KPI
Annual Meta-assessment Process
The Office of Academic Planning and Assessment will utilize a locally developed rubric designed to evaluate the overall quality of a program's annual assessment plans to facilitate an annual review of assessment plans stored within the Online Assessment Tracking Database (OATDB). The results of this evaluation should indicate that 80%, or more, of the reviewed assessment plans for each College/Division reviewed should be rated as “Acceptable” or better. Additionally 80%, or more, of the total number of assessment plans reviewed from across the University should be rated “Acceptable” or better.
Result
Results From The 2014-2015 Meta-assessment Review
During the 2014-2015 assessment cycle, the Office of Academic Planning and Assessment oversaw the Meta-assessment review of assessment plans from all seven of the Academic Colleges. Four of the seven academic colleges conducted college-led meta-assessment reviews of their units. Units within the remaining three colleges were reviewed by the Director of Assessment. Additionally, final Meta-assessment Summary Reports were ultimately received from six of the seven colleges.
A summary of results are provided here for each of the seven colleges. Percentages represent the percentage of acceptable and exemplary assessment plans/elements from each college:
College 1 – Self-Reviewed Overall – 95.24% Goals - 100% Objectives - 100% Indicators – 87.50% Criterion – 93.75% KPIs – 87.50% Findings/Results – 78.26% Actions – 69.57% Plan for Continuous Improvement Update – 72.73% New Plan for Continuous Improvement – 65.22%
College 2 – OAPA Reviewed Overall – 43.75% Goals – 25% Objectives – 93.75% Indicators – 72.73% Criterion – 81.82% KPIs – 100% Findings/Results – 100% Actions – 43.75% Plan for Continuous Improvement Update – 92.86% New Plan for Continuous Improvement – 31.25%
College 3 – OAPA Reviewed Overall – 48.28% Goals – 72.41% Objectives – 93.10% Indicators – 95.83% Criterion – 100% KPIs – 71.43% Findings/Results – 82.76% Actions – 51.72% Plan for Continuous Improvement Update – 82.14% New Plan for Continuous Improvement – 42.31%
College 4 – Self-Reviewed Overall – 100% Goals – 100% Objectives – 100% Indicators – 100% Criterion – 93.33% Findings/Results – 93.33% Actions – 86.67% Plan for Continuous Improvement Update – 86.67% New Plan for Continuous Improvement – 78.57%
College 5 – OAPA Reviewed Overall – 25% Goals – 100% Objectives – 100% Indicators – 93.75% Criterion – 93.75% KPIs – 83.33% Findings/Results – 78.95% Actions – 31.58% Plan for Continuous Improvement Update – 73.68% New Plan for Continuous Improvement – 31.58%
College 6 – Self-Reviewed Overall – 92% Goals – 88.46% Objectives – 92.31% Indicators – 82.61% Criterion – 81.82% KPIs – 68.75% Findings/Results – 84.62% Actions – 84% Plan for Continuous Improvement Update – 80% New Plan for Continuous Improvement – 76.92%
College 7 – Self-Reviewed Overall – 78.05% Goals – 97.56% Objectives – 100% Indicators – 93.94% Criterion – 78.13% KPIs – 78.57% Findings/Results – 79.49% Actions – 75.61% Plan for Continuous Improvement Update – 70.73% New Plan for Continuous Improvement – 70%
Overall for all Academic Colleges Overall – 69.46% Goals – 85.80% Objectives – 97.06% Indicators – 90.58% Criterion – 88.24% KPIs – 78.95% Findings/Results – 83.83% Actions – 64.88% Plan for Continuous Improvement Update – 78.05% New Plan for Continuous Improvement – 58.54%
Overall for Self-Reviewed Colleges Overall – 88.24% Goals – 96.15% Objectives – 98.10% Indicators – 90.80% Criterion – 84.71% KPIs – 76.32% Findings/Results – 82.52% Actions – 77.88% Plan for Continuous Improvement Update – 75.73% New Plan for Continuous Improvement – 71.84%
Overall for OAPA Reviewed Colleges Overall – 40% Goals – 69.23% Objectives – 95.38% Indicators – 90.20% Criterion – 94.12% KPIs – 84.21% Findings/Results – 85.94% Actions – 43.75% Plan for Continuous Improvement Update – 81.97% New Plan for Continuous Improvement – 36.07%
These results revealed several areas for institutional improvement. No individual college exceeded 80% for all assessment plan elements. Generally, the following elements saw the greatest weakness: Overall Score, KPIs, Actions, Updating Progress on the Previous Cycle’s Plan for Continuous Improvement, and the New Plan for Continuous Improvement. Differences were also observed between the colleges that conducted their own meta-assessment reviews and those that were reviewed by OAPA. In general, OAPA reviews were lower than those conducted by the colleges. This would indicate that more work is needed to improve the quality of the college-led meta-assessment processes to ensure that they are meaningfully examining the quality of their assessment plans. A complete breakdown of the meta-assessment results can be found in the attached file.
Action
Improvements To The Meta-assessment Process
During the 2015-2016 cycle, OAPA will work to continue to formalize meta-assessment across campus. The Director of Assessment will work with each of the Academic Colleges to ensure that each college conducts their own locally-led meta-assessment review. Additionally steps will be taken next year to provide more feedback to the colleges regarding the quality of their meta-assessment reviews. OAPA will institute a review process in which a sample of scored rubrics will be evaluated for accuracy and consistency. These results will then be given to the colleges to provide them with feedback regarding the accuracy of their scoring.
Objective
Provide Quality Assessment Support Resources
The Office of Academic Planning and Assessment will provide quality assessment resources to the University community through its website, ongoing training sessions, and workshops.
KPI
Website Tracking
Utilizing Google Analytics, the Office of Academic Planning and Assessment will track traffic coming to the department's website. Data from the 2014-2015 cycle will serve as a baseline for subsequent assessment cycles.
Result
Google Analytics Results For The Office Of Academic Planning And Assessment
Google Analytics reports were compiled for each month of the 2015 Fiscal Year for the Office's homepage. A summary of the key results are provided below:
Direct comparisons are available for the months of June, July, and August. The month of June saw increases from the previous fiscal year of 219% for sessions, 262% for users, and 68% for page views. The month of July saw increases from the previous fiscal year of 401% for sessions, 460% for users, and 225% for page views. The month of August saw increases from the previous Fiscial year of 114% for sessions, 124% for users, and 56% for page views.
KPI
Number Of Workshops/Training Sessions Held
The Office of Academic Planning and Assessment will conduct at least 100 workshops/training sessions related to the annual assessment process being conducted at SHSU. These sessions may range from large, group workshops to individual training sessions.
Result
Total Number Of Meetings, Workshops, And Training Sessions Fo FY15
For the 2015 FY, the Office of Academic Planning and Assessment did not meet its Objective of 100 meetings, workshops and training sessions. Only 76 total events were identified by the Director of Assessment. broken down as follows:
32 small group assessment meetings (decrease from the previous year) 16 meetings related to Core Curriculum Assessment 8 assessment workshops (increase from the previous year) 7 small group OATDB training sessions 6 meetings related to the implementation of CampusLabs 3 meetings related to the SACSCOC 5th Year Interim Report 2 meetings related to assessment mini-grants 1 meeting related to Meta-assessment 1 meeting related to Administrative Program Review
KPI
Workshop Evaluation Survey - Satisfaction
Training session attendees will complete a brief survey, consisting of three Likert-scale questions and three open-response questions, which indicate their satisfaction with the services provided by the Office of Academic Planning and Assessment. A copy of the survey is provided as an attachment. The average response to the Likert-scale question related to participant satisfaction should be 4 or higher, indicating that they were satisfied with the services provided by our Office. Additionally, respondent comments from the three open-response questions should be generally positive.
Result
Workshop Satisfaction Results
Workshop evaluation surveys were completed for 5 workshops during the 2014-2015 assessment cycle. The average satisfaction score for each workshop exceeded 4, indicating that respondents were satisfied with the training provided from our office. Responses to the qualitative questions were similarly positive.
International Programs Assessment Workshop (9/5/2014) - 4.33
College of Criminal Justice, Victim Services Managemenent Assessment Committee Workshop (12/3/2014) - 4.57
College of Health Sciences Workshop (12/15/2014) - 4.67
Residence Life Workshop (4/8/2015) - 4.44
College of Health Sciences Workshop (4/15/2015) - 5
Please see the attached reports for additional information.
KPI
Workshop Evaluation Survey - Confidence
Training session attendees will complete a brief survey, consisting of three Likert-scale question and three open-response questions, which indicate their satisfaction with the services provided by the Office of Academic Planning and Assessment and their confidence with assessment practices. A copy of the survey is provided as an attachment. 90% of respondents will report a pre-to-post increase in their confidence with regards to implementing effective programmatic assessment. Additionally, 90% of respondents reporting a post-score of 4 or 5, indicating they are now confident or very confident.
Result
Assessment Confidence Results
Workshop evaluation surveys were completed for 5 workshops during the 2014-2015 assessment cycle. In none of the workshops did 90% of the respondents report a pre-to-post increase. Additionally, in only 3 of the 5 workshops did 90% or more of the respondents indicate they were confident or very confident with their ability to design and implement effective programmatic assessment. However, averages did increase, pre-to-post, for three of the five workshops. In two workshops averages decreased, although this could be explained by very high pre-averages, meaning that those attendees may now have had a more accurate perception of their own abilities.
Please see the attached reports for a complete breakdown of the results for each workshop.
Action
Improving The Quality Of Assessment Training And Support
In general, OAPA staff are very pleased with what the office has been able to accomplish with regards to training and support, given limits to both time and staff resources. Analysis of monthly web hits show that the OAPA website is seeing increased web traffic, which means we are increasing our visibility. The office did not meet its objective of holding 100 meetings, workshops, or training sessions, but did increase the number of assessment workshops it held. OAPA will be expanding its staff in 2015-2016, which should allow the office to conduct more workshops and meetings with campus constituents. Finally, office staff are also pleased with the results gained from the workshop evaluation surveys, In general, constituents are pleased with the quality of our training sessions. We did not meet the objectives related to increasing confidence with regards to assessment; however, this was because clients often reported high levels of confidence to begin with. At times, a decrease in confidence may be appropriate if clients have a much more realistic view of their abilities after a training session. Similarly, it should not be seen as a negative if a client maintains their existing level of confidence. Therefore, OAPA staff will look to revise this objective to better reflect what was observed from the data this year.
Goal
Support And Facilitate The Undergraduate Program Review Process
The Office of Academic Planning and Assessment will support and facilitate the Undergraduate Program Review Process as Sam Houston State University.
Objective
Facilitate A Quality Undergraduate Program Review Process
The Office of Academic Planning and Assessment will facilitate a quality undergraduate program review process through the creation of formal policies and guidelines, program review templates, and acquisition of program-specific data. Such resources will be made available through the Office website and print materials.
KPI
Undergraduate Program Review Guidelines
Development and approval of formal policies and guidelines for Undergraduate Program Review.
Result
Undergraduate Program Review Process
Work related to this Objective is still on-going. Reorganization within the Division of Academic Affairs, and staff-duty changes within the Office of Academic Planning and Assessment prevented more progress from being made in the development of guidelines for the Undergraduate Review process.
Action
New Staff Member To Be Added For 2015-2016
Given the reorganization that has taken place within Academic Affairs, and the merger of new roles and duties within OAPA, our office will be hiring an additional staff member during 2015-2016. This new staff member will be involved with assessment, and will also oversee the Undergraduate Program Review Process. One of the first priorities for this new staff member, when hired, will be to finalize the Undergraduate Program Review Guidelines and Process.
Goal
Support The Institution's Ongoing Southern Association Of Colleges And Schools Commission On Colleges (SACSCOC) Accreditation Efforts
The Office of Academic Planning and Assessment will support the institution's ongoing efforts to respond to all SACSCOC requirements for maintaining accreditation.
Objective
Facilitate Completion Of The SACSCOC Fifth-Year Interim Report
The Office of Academic Planning and Assessment will work with the University community to ensure the successful completion of the SACSCOC Fifth-Year Interim Report. To this end, the Office will work to disseminate information and resources, provide necessary training, and complete and submit all required documents.
KPI
Prepare A Quality And Thorough Compliance Narrative Document For The 5th Year Interim Report
The Office of Academic Planning and Assessment will work with university personnel to ensure that a thorough, accurate, and quality compliance narrative document is prepared for the SACSCOC 5th Year Interim Report.
Result
5th Year Report Submission And Response
In March 2015, SHSU submitted its SACSCOC Fifth Year Interim Report. In July 2015, the institution received the SACSCOC findings associated with its 5th Year Report submission. SHSU was required to submit a Referral Report addressing only one SACSCOC standard. The extremely limited findings of the SACSCOC reviewers indicate that SHSU successfully submitted a thorough, accurate, and quality compliance narrative document. The requested Referral Report was submitted in September 2015.
Action
Fifth Year Report Complete
There is no additional action required for the SACSCOC Fifth Year Interim Report.
Objective
Ensure Institutional Compliance With And Timely Submission Of Required SACSCOC Documentation
The Office of Academic Planning and Assessment will work with the University administration to ensure that all required SACSCOC documents are submitted timely, and appropriately.
KPI
Appropriate Submission Of SACSCOC Required Documentation
The SACSCOC liaison, and the Office of Academic Planning and Assessment, will ensure that all required SACSCOC documents, such as Institutional Profiles, Letters of Notification, Prospectuses, Institutional Profiles, etc., will be summited timely and appropriately to the SACSCOC.
Result
Enrollment And Financial Profiles
SHSU submitted annual Enrollment Profiles and Financial Profiles, as required by SACSCOC, by stated deadlines.
Result
Substantive Change Reporting
SHSU submitted substantive change notifications and prospectus’ to SACSCOC, as per required timelines, for applicable changes implemented in the 2014-2015 academic year. SHSU also developed a Substantive Change Training and Education Program to be implemented in the 2015-2016 academic year.
KPI
Address Functional Deficits In Faculty Credentials Reporting System
Following the institution's conversion from a 'home-grown' ERP system to Banner, some functionality relating to Faculty Credentials reporting was lost. Steps will be taken to correct functional aspects of programming to include centralizing faculty degree entry into Banner and alteration of existing reports to align with new Banner structure.
Result
Faculty Data Transition Into Banner
During the 2014-2015 academic year, progress was made in the extensive work required to transition all faculty-related data into Banner. As of September, 2015, the Office of Academic Planning and Assessment has worked closely with the Office of the Provost, IT@Sam, Institutional Effectiveness, Graduate Studies, and the Registrar's Office to identify Banner fields for all faculty data functionality and reporting. Much progress has been made with regard to standardizing degree and major awards for faculty and identifying inconsistencies with existing FICE codes.
Related to the Faculty Credentials Reporting System, several technical improvements were made during the 2014-2015 academic year. Improved user functionality was added for the uploading of faculty vita and syllabi, improved administrative permissions management was added, and corrections were implemented to reflect accurate course listings.
Action
Implementation Of Substantive Change Training And Education Program
During the 2015-2016 academic year, the Office of Academic Planning and Assessment will fully implement the Substantive Change Training and Education Program. At least once during the academic year, the SACSCOC Liaison will meet with the Council of Academic Deans and the President's Cabinet to provide updated training and education related to substantive change reporting requirements.
Action
Preliminary Timeline Development For SACSCOC 10 Year Reaffirmation
During the 2015-2016 academic year, the Office of Academic Planning and Assessment will work to develop a preliminary timeline for the institutions next 10-year Reaffirmation of Accreditation Review. In addition, the SACSCOC Liaison will work with the TSUS office to ensure SHSU is prepared to meet revised audit expectations as part of the 10-year report.
Action
Completion Of Faculty Data To Banner Conversion
During the 2015-2016 academic year, the institution will complete its work to transition faculty data into Banner. Related updates to the Faculty Credentials Report will also be completed.
Goal
Support The Strategic Planning Process For The Division Of Academic Affairs
The Office of Academic Planning and Assessment will support the ongoing strategic planning process underway within the Division of Academic Affairs.
Objective
Provide Quality Strategic Planning Resources And Processes
The Office of Academic Planning and Assessment will provide quality strategic planning resources and facilitate effective planning processes within the Division of Academic Affairs.
KPI
Facilitate Development Of A Comprehensive And Quality Academic Affairs Strategic Plan
The Office of Academic Planning and Assessment will facilitate strategic planning discussions within Academic Affairs, providing the necessary resources and structure to the process. Planning meetings and retreats will be scheduled and data resources provided as needed.
Result
Strategic Plan Development Delayed
Due to constraints on academic administrator personnel availability in the summer of 2015 and extensive, ongoing divisional projects, a decision was made to delay the strategic planning meetings until the following academic year. Preliminary planning meetings were held throughout the spring semester in which basic plan components were identified. Data requests were made and have been partially fulfilled. Work continues on annual operational planning and connection to the budgeting process.
Action
Strategic Planning Consultant
Preliminary strategic planning discussions led to a realization that the process would be aided by an external strategic planning consultant. The committee proposed a consultant name and meetings will be scheduled to discuss project scope and deliverables in the 2015-2016 academic year.
Goal
Promote The Scholarship Of Assessment
The Office of Academic Planning and Assessment will promote the growing scholarship of assessment, within SHSU, Texas, and the nation, through research, presentations, and publications.
Objective
Scholarly Presentations And Publications
The Office of Academic Planning and Assessment will make presentations and submit publications on various assessment related topics through state, regional, and national venues.
KPI
Scholarly Presentations
The Office of Academic Planning and Assessment will track the number of scholarly presentations conducted by members of its staff for the 2013-2014 assessment cycle. The minimum target for success will be 4 presentations at state, regional, or national conferences or meetings.
Result
Scholarly Presentations During 2014-2015
During the 2014-2015 assessment cycle, members of the Office of Academic Planning and Assessment made a total of 6 presentations at state, regional, or national conferences or meetings. This total exceeded the expected target of 4. Presentations by OAPA staff are outlined below:
Roberts, J. (2015, February). Building global citizens: One institution’s plan to use multiple measures to assess social responsibility, intercultural knowledge, and civic engagement. Presented at the 15th Annual Texas A&M University Assessment Conference, College Station, TX.
Roberts, J. (2015, February). Promoting the scholarship of assessment at Sam Houston State University through mini-grants. Poster presented at the 15th Annual Texas A&M University Assessment Conference, College Station, TX.
Simon, J. F., Roberts, J., Bledsoe, E. C. (2015, February). Assessing the assessment: Utilizing rubrics to evaluate the quality of programmatic assessment plans. Workshop presented at the 15th Annual Texas A&M University Assessment Conference, College Station, TX.
Roberts, J., Bledsoe, E. C., & McLawhon, R. (2014, October). The development and evolution of assessment review processes at two Texas institutions. Presented at the 2014 Assessment Institute, Indianapolis, IN.
Roberts, J. (2015, February). Maturation of assessment practices for states within the Southern Association of Colleges and Schools accreditation region. Paper presented at the Southwest Educational Research Association Conference, San Antonio, TX.
Franklin, S., Bledsoe, E. C., Roberts, J (2015, January). Two universities approach to core curriculum and meta-assessment. Invited presentation at the Texas Council of Chief Academic Officers Annual Retreat, Austin, TX.
KPI
Scholarly Publications
The Office of Academic Planning and Assessment will track the number of scholarly articles submitted and accepted for publication by member of its staff. As this is a new measure the minimum target for success will be one article submitted and accepted for publication, per year.
Result
No Scholarly Publications Submitted For 2014-2015
Although several potential articles are in the works, the staff within the Office of Academic Planning and Assessment submitted no scholarly works for publication during the 2014-2015 assessment cycle. The primary reason for this was a lack of available time. Because of the numerous other tasks of the office (e.g., submission of the SACSCOC 5th Year Interim Report, Core Curriculum Implementation, ongoing programmatic assessment), and the lack of staff resources.
Action
Increasing Scholarly Publications
The Office of Academic Planning and Assessment is very satisfied with the number of scholarly presentations being made by office staff; however, the Office remains disapointed that they were not able to publish any scholarly articles related to the assessment of student learning. Although office staff, principally the Director of Assessment, are actively engaged in research related to this subject, other office duties have prevented them from writing up and submitting the results of this research for publication. With the successful completion of the SACSCOC 5th Year Interim Report, and the addition of new office staff, it is hope that the office will be able to devote more time and energy towards publishing some of the research conducted by the Office.
Objective
Assessment Mini-Grants
The Office of Academic Planning and Assessment will help promote the scholarship of assessment at SHSU through sponsorship of assessment mini-grants. These grants are available to faculty and staff at SHSU to help fund new or ongoing assessment practices within programs, offices, or departments; or to help fund travel to make assessment-related presentations at professional conferences.
KPI
Grant Awards For 2014-2015
The Office of Academic Planning and Assessment will award 5 $1,000 Assessment Mini-Grants during the 2014-2015 assessment cycle. At the completion of each grant-funded project, each recipient will also complete and submit a follow-up report.
Result
Total Assessment Mini-Grants Awarded In 2014-2015
Expectations for the assessment Mini-Grants were exceeded. A total of 12 applications were received, and funds were available to award 10 $1,000 grants. At this time, the Office of Academic Planning and Assessment is still waiting to received follow-up reports from all of the grant recipients. Grant recipients were as follows:
Julie Albert and Linda Zientek, Department of Mathematics and Statistics - Student Success Rates of Developmental Mathematics Students By Student Profiles
Maria Holmes, Honors College - Honors College Admission and Retention Assessment
John Jordan, SAM Center - Qualitative Program Assessment
Steven Koether, College of Sciences - FoS: Online Peer Evaluation of Effort - Improving the Ease of Use
Valerie Powell, Department of Art - Assessing Risk-Taking in the Classroom
Todd Primm, Department of Biological Sciences - Developing the Microbiology Concept Inventory
Glenn Sanford and David Wright, Department of Psychology and Philosophy - Assessing Student Metacognition as an Indicator of Critical Thinking Skills
Jesse Starkey, Global Center for Journalism and Democracy - International Journalism Training Follow-up Assessments
James Van Roekel, Lowman Student Center - LSC Student Evaluation
Tamara Waggener, Department of Political Science - Assessment and Pedagogy in Required POLS Course
Action
Formalization Of The Assessment Mini-grant Program
Given the success experienced with the Assessment Mini-Grant Pilot Program, the Office of Academic Planning and Assessment will formalize this program for the 2015-2016 cycle. The Office has requested, and received, formal funds to devote to this project and will be able to provide up to 10 assessment mini-grants for 2015-2016. OAPA will seek to establish formal dates for the submission and review of applications and for the rewarding of grants. This process will be formalized during 2015-2016.