OATdb Archive

2012 - 2013

Institutional Effectiveness

Goal
Collect, Analyze And Disseminate Institutional Data And Relevant Information In An Accurate, Timely And Understandable Manner
Collect, analyze and disseminate institutional data and relevant information in an accurate, timely and understandable manner

Objective
Develop And Maintain Outcome Indicators
Develop and maintain a core of outcome indicators at multiple institutional levels.


KPI
Data Definitions
Create operational document identifying key data definitions.

Result
Data Cookbook
It was determined that the institution owns a software license for a product that will allow IE to facilitate the development and documentation of Cognos report and Banner term definitions.  IE has coordinated meetings with IT and Banner module functional users to create a framework in Data Cookbook for the population of this information.  This KPI was originally intended to include a single document; however, the process of planning for the implementation of Data Cookbook has resulted in the creation of a number of supplemental documents.  A process manual has been developed and IE is underway in posting the information to the appropriate sections of the software product.  Also, an inventory of existing Cognos reports is underway to identify those items that need to be included in the Data Cookbook product.  Future objectives are to import the majority of active Cognos reports in the Data Cookbook and all the terms included within those reports so that users can search terms and reports and determine independently what resources currently exist to address reporting needs.  The conversion is expected to take 6 to 12 months.

Action
Report And Term Construction And Import
IE staff will construct import files that consist of Cognos report details and associated terms for import into Data Cookbook. Public reports will be prioritized over non-public reports.  Future objectives are to store the majority of active Cognos reports in the Data Cookbook and all the terms included within those reports.  The conversion is expected to take 6 to 12 months.


Objective
Increase The Use Of State And National Data Sources In Institutional Analysis And Presentation
Increase the use of state and national data sources in institutional analysis and presentation.

KPI
Utilize THECB, IPEDS And Common Data Set Information.
Integrate state THECB and national IPEDS and CDS data into work orders where data source is appropriate.

Result
Using Data From THECB, IPEDS And CDS
The majority of data requests submitted to IE are for institutional data aggregated at levels and in groups that are not similar to the aggregations made by THECB, IPEDS or CDS.  A review of the 2012-2013 IE work order log reveals approximately 10 of 350 work orders were suitable for inclusion of data from one of these sources.

Action
Remove Item From Assessment Focus
As the majority of data requests submitted to IE from SHSU users are for unique or custom institutional data that is not reported to state or national agencies, the opportunity to use pre-defined variables in meeting local needs is limited. For this reason, this item will be removed from future assessment focus. IE will continue to pull standardized data when possible; however, the requests will drive this activity.  For those that may be conducive to the inclusion of other data sources, IE staff will develop an inquiry process with requestor to determine if national benchmarking would improve the data interpretation.


Goal
Contribute Materially In The University-wide Process For Continuous Improvement By Assisting Administrative Units In The Evaluation Of Operations
Contribute materially in the university-wide process for continuous improvement by assisting administrative units in the evaluation of operations

Objective
Develop Administrative Program Review Processes And Procedures
Design, develop and test processes, procedures and forms for the implementation of institutional Administrative Program Review.

KPI
Administrative Program Review Processes
Develop forms and procedures required to communicate with administrative departments to accomplish the process of administrative program review.


Result
Development Of Key Program Documents
Prior to the retirement of the staff member developing the Administrative Program Review program, the following forms and documents were designed for implementation of the APR at SHSU: Nominees for APR Committee Pool; Instructions for APR Committee; APR Evaluation Rubric; APR Handout; APR Set-up and Main Process Narratives; APR Preparation Survey.  These documents were structured from a review and evaluation of US universities with an APR program.

Action
Fill Position And Pursue Program Implementation And Pilot
Hiring is underway to fill this position.  Once this occurs, the Assistant Director for Administrative Program Review will finalize pending program documents and begin a pilot APR at SHSU.

Goal
Improve Office Of Institutional Effectiveness Capacity (Two Year Time Frame)
Improve Office of Institutional Effectiveness work order management and fulfillment strategies so that measurable improvement in quantity and user satfisfaction occur.

Objective
Track And Evaluate User Satisfaction
Increase the overall satisfaction level by users with output.

KPI
Post Fulfillment Survey
Implement contact with users to evaluate satisfaction with output.

Result
Results Of Satisfaction Survey
Institutional Effectiveness issued an invitation via email to individuals who requested data through the IE work order process from Fall 2012 – Summer 2013.  The results of the survey were compared to the results of the same survey issued for Fall 2011 – Summer 2012.  In terms of positive and negative changes by topic or question, the results are as follows:

A. Level of satisfaction with the time required to respond and to complete – Significant improvement

B. Level of satisfaction with the clarity of written explanations – Significant improvement

C. Level of satisfaction with the thoroughness of written explanations – Slight improvement

D. Did you receive the information you requested? - Slight improvement

E. Level of satisfaction with the clarity and thoroughness of the oral explanation - Improvement

F. Level of satisfaction with the presentation style or format – Slight reduction

G. Level of satisfaction with the usefulness of the response/data to your project/work – Slight reduction

H. Level of satisfaction with the customer service - Reduction


Action
Increase Communication With SHSU Users After WO Completion
Customer satisfaction survey results indicate SHSU Users need their data presented in a manner that is easier to understand and to apply to their project.  Due to the indication that the presentation style and format are not meeting expectations, IE staff will contact users after delivery of data to determine what is lacking in the specific report, if anything.  Additionally, IE staff will inquire as to how IE can improve the User's satisfaction upon delivering the report.  Lastly, a request will be added to the satisfaction survey for Users to suggest a manner in which their satisfaction would increase.  IE staff have increased the attention to post-report communication, and the indication that satisfaction has fallen is in contrast to that effort.

Update to previous cycle's plan for continuous improvement

I. A result of the reorganization of April 2012 was the reassignment of the OATDB reporting responsibilities from the previous director to the new department head.  Issues of operational process monitoring and documentation that were determined to be of significance and needing correction were inconsistent handling of all stages of the work order process, lack of sufficient validation of data output or proofreading of final reports, poor quality of communication with requestors (multiple typographical errors in communication; long, complicated email correspondence that confused readers; unorganized and redundant filing of digital files within internal directories preventing ease of access; etc.), and assignment of work order responsibilities to inexperienced International graduate students who possessed limited communication skills.  Initial correction included establishing a defined protocol for the receipt, logging and assignment of work order requests.

(A) Designate a single individual to initiate each work order as a single point of contact in order to promote consistently applied rules, judgments and interpretations of information capture.  This process requires the Administrative Assistant to be the primary individual to log new work order requests in the department log and name a repository folder in the IR directory for subsequent files.

(B) Establish staff review of work order status, regularly sharing brief oral reports of time invested in completing each work order and noting important changes and developments.  The Assistant Director of IR (ADIR) reviews and updates the log information as needed, utilizing OneNote for more extensive details, and proceeds to assign the task and deadline to one or more IR Analysts.  Communication with the requestor is coordinated and documented by the Analyst with the ADIR.  Longitudinal tracking studies are typically assigned to one particular Analyst for consistency.  The final report is proofread by the Analyst, ADIR and the Assistant VP for IE, depending on the level of complexity and the need for evaluation of clarity and completeness.  Upon submitting the finished report to the requestor, the ADIR sets a time in the near future to call or email to determine if there are questions about the results.

(C) Develop and implement an organizational schema for a work order table that will permit the documentation of jobs within jobs, i.e., the hierarchical or familial linking of a new job to the original job from which it evolved as a variation.  New tasks associated with an existing or recent work order are added and numbered as a sub-order in the log.

To promote further consistency and professionalism, the department staff developed a template for use in creating tables, which will ultimately be submitted to a requestor.  Color palette, header and footer details were identified, as well as minimum criteria for abbreviations, spelling of terminology and use of hyphens in words that may or may not be hyphenated.  With uniformity and a clear intake process, as well as an emphasis on quality of data and report logic, the modifications in FY2013 are expected to increase the confidence SHSU users have in the department of Institutional Effectiveness.

(D) Implement an automated, database driven, searchable work order documentation system linked to a web-based input page as permitted by funding opportunities or by sharing resources with IT.  The recent implementation of Cherwell work order system and Data Cookbook in FY2013 will provide additional opportunities for this objective.  Future review and analysis will be required.


 II. The Office of Institutional Effectiveness gathers anonymous input from individuals who request data via an online survey.  Refining and improving the solicitation of client feedback and potential client needs will help focus efforts on particular areas in need of improvement.

(A) Administration of Client Satisfaction Surveys with tokens in order to identify feedback with corresponding work orders.  Due to technological issues, the customer satisfaction surveys conducted by IE did not include tokens with which to identify responders.  The intention of tying the feedback to a work order was to better understand complaints or criticism.  Based on the improvements in areas of satisfaction as identified in the most recent survey, this item may not need to be implemented.  Additional fields will be included on the survey to encourage constructive and specific criticism.

(B) Administration of Client Satisfaction Surveys near the end of each term (Fall, Spring, Summer) , if not immediately following the delivery of the completed work order in order to increase the reliability and specificity of work order related feedback.  The survey was administered at the end of Fall 2012 and Summer 2013.  Due to new processes that were implemented in the work order system, assessing the changes in Spring 2013 seemed too early.

(C) Administration of strategic needs assessments re: underutilized IE services or services which may be anticipated to play a stronger role in supporting a strategic direction in which the University may move.



  1. Encouraging the habit of regular follow-up contact with completed work order clients will provide the IE staff the opportunity to offer highly individualized explanation and assistance in interpretation of delivered results. FY12 Client Satisfaction Survey results suggest written explanations were less clear and thorough than oral explanations, which did not appear to be frequently offered.  Regular and timely follow up by email and phone call was initiated in FY2013.

  2. In order to help assure that IE staff are prepared to meet the challenges of mastering new and changing technology, analysis tools and applications, provide adequate time and opportunity for formal training, including time for practice, self-teaching and peer-tutelage.  IE staff attended training in SQL, Banner, Excel, Cherwell, PeopleAdmin, and Cognos in FY2013.  This represents more extensive training than the staff have received in the past several years.  Staff are encouraged to attend any and all SHSU IT training.

Plan for continuous improvement In reviewing the primary goals of the Office of Institutional Effectiveness and the manner in which they are strategically aligned with the Division of Enrollment Management goals, several processes underway are clearly in their infancy and need further development.  The primary goals are 1) Collect, Analyze And Disseminate Institutional Data And Relevant Information In An Accurate, Timely And Understandable Manner; 2) Contribute Materially In The University-wide Process For Continuous Improvement By Assisting Administrative Units In The Evaluation Of Operations; and 3) Improve Office Of Institutional Effectiveness Capacity.  Actions associated with Goal 2 are contingent on hiring a staff member to finalize development and implement Administrative Program Review.  For this reason, assessing the status of this goal is problematic at this time, although it is a vital part of the operation of IE, once it begins.  For that reason, it will not be covered further in this annual review.  The remaining goals can be better addressed in terms of a plan for improvement.

 

Intrinsically, the Institutional Research function includes collecting, analyzing and disseminating data, but the speed and satisfaction of users associate with this task are the focus of efforts to improve.  The IE staff will be advised and trained on the elements of the satisfaction survey to increase accountability and understanding.  At monthly staff meetings, staff will be asked to rate themselves and offer internal perspectives on work orders that have the potential for user confusion or those that may seem to deviate from the intentions of the user.  To better understand the users’ intentions in the work order process, IE has expanded the work order form and provided numerous examples of data types that can be accessed for users.  This is the current plan to improve the results of the satisfaction surveys.

 

Additionally, the SHSU user community (internal and external) are often limited to accessing institutional data and understanding the reports currently available.  IE has collaborated with subject matter experts on campus in the past 12 months to initiate the Data Cookbook program.  For the near future, IE will continue populating the online product with reports and terms to assist users in meeting their reporting needs.  IE will continue collaborating with IT for large report construction and subject matter experts and data owners for modification of existing reports and the creation of smaller reports, all in the Cognos report writing environment.  It is anticipated that IE can remove the burden of individual report construction from departments and centralize the storage of information related to reports and terms.

 

Since the reorganization of the IE role in April 2012, a reporting position was filled and recently vacated again.  In order to increase the capacity of work performed by the IE staff, empty positions must remain filled with appropriate personnel.  To better accommodate confusing language in the position description and title of the reporting position, recently the title was changed from Administrative Coordinator, Senior to External Reporting Specialist.  The position posting was expanded to include more specific daily duties and basic skills required to meet proficiency standards.  Lastly, an exercise in Excel and dataset proficiency will be introduced into the hiring process to improve the likelihood the new hire has all the skills needed to start with little training in that area.  This was an obstacle faced by the previous incumbent. 

Sam Houston State University | A Member of The Texas State University System