Continally improve the support services to internal constituents

As one portion of its primary responsibilities, IRA must comply with federal & state government, as well as with non-governmental reporting mandates on behalf of the university and its various functions. Mandated activities include providing information, conducting research studies, analyzing, interpreting and reporting data. It is critical that all of these mandated jobs be performed to specification and within deadline, lest the university suffer dire consequences.

The Office of Institutional Research and Assessment will use the IRA Workorder Table to track when mandated jobs are received by the IRA Office.  Mandated jobs may include any official jobs for government entitites such as IPEDS, THECB, TSUS, Board of Regents, Legislative House or Senate bill compliance reports, LBB, LAR, U.S. Dept of Education, NSF/NIH reports, etc.  Mandated jobs may also include mandatory, non-governmental jobs such as the national surveys by Peterson's, College Board,  US News and World Reports, SACS, ACT, VSA, Business Week,  etc.  Incoming jobs are recorded and progress tracked through completion in the IRA Workorder table.  Other data captured in the IRA WO table for a given FY include: Whether mandated (external or internal), cient requested , internal IRA project, external to University (but not mandated); priority level; to whom job is assigned; when completed; and the shared drive locations of job-related files.  The 2009-2010 Work Order Table will be reviewed, mandatory research and reports will be identified among the work orders and they will be counted  along with completion times, and the number will be reported in relation to timely delivery.

100% of mandated reports received by the IRA office will be completed.

Based upon internal IRA accuracy checking and completion tracking systems, 100% of mandated jobs have been accurately completed and submitted by deadline, in FY 2010, through June 31. . FY10 mandated jobs included:
39 total Government Mandated Jobs - US Federal or TX State
42 total Non-Govt Mandatory national survey jobs

DETAILS - MANDATORY GOVT JOBS  
1 – Repeat all analyses and calculations for Government Report to produce verification of previously reported results
6 – Government report follow ups requiring repeating analyses and calculations with variations on definitions or specifications
6 – Government Report Data Request conveyed by university executives in support of legislative questions
7 – Government Survey Request
19 – Government Reports

The assessment of our efficiency in handling mandated work could be improved if all sub jobs and variations on the same mandated jobs were recorded in the IRA Work Order Table as separate work order jobs. Presently, many non-fixed mandated jobs will have extensive add-ons or re-workings under different criteria and other variations requested while the original job is still in preparation. Often these extensions are NOT recorded.  Consequently our ability to accurately gauge how long we spend and how heavily mandated jobs draw on our resources is compromised.  The director plans to inform the staff about the need for a carefully followed policy practice of documenting every job extension as a separate job or both mandated and non-mandatory jobs.

The automation of our work order table through a database fed by the web work order template would greatly facilitate the execution of this policy. We know that manual work order entry is cumbersome and time consuming and encourages forgetfulness, at very busy times, about entering job information which may enter unexpectedly by phone, email or walk-in requests extension requests. The IRA Office group is also discussing backing the IRA WO web template, or an expanded version thereof, with a database. This discussion requires permissions and approvals from Computer Service/IT and it has been rejected a number of times in the past. We are discussing and working on alternatives that might be acceptable to CIS/IT.

Provides technical assistance in validating and documenting unit and program assessment processes in order to achieve full compliance for the reaffirmation of accreditation by the Southern Association of Colleges and Schools and other professional accreditation groups.

Although he IRA Office assists with a number of other accreditation efforts that occur periodically on campus, for purposes of this FY10 self assessment, The IRA office will focus upon assistance rendered in SACS Reaffirmation Assessment Monitoring Effort. We will count only evidence pertaining to substantive work performed for SACS assessment coaching and Institutional Effectiveness Committee support. 

1. We will count  pertinent  coaching notes and communications delivered via e-mail; sign-in sheets at group coaching sessions; IRA Workorder Request Table entries;  coaching assignments made in 2010 by the Institutional Effectiveness Committee; coaching assignments made by the SACS Liaison. 
2. Data will be assessed for "substantive" nature by number and duration  of help instances , intensity and types of assistance provided in connection with the SACS reaffirmation process. These data are found  in  Institutional Effectiveness _Coaching Outlook Mail Program Folders and in shared computer drive folders devoted to Institutional Effectiveness\FY 10 Coaching.
3. The data in these folders will be reviewed, classified, listed to a spreadsheet  and tallied to determine the number of units substantively assisted. (attached)
4. The number of  units judged to be compliant  by the SACS Liaison, IE Committee Coaches  and the IRA Office agreement when the SACs Monitoring Report is in final compilation will be used to calculate the percentage of complete and  compliant units among all those that were substantively assisted by the IRA Office. 


(Assistance to a unit is defined as follows:
The OAT DB assessment reporting entries of all assisted units will be evaluated using a locally constructed rubric and scoring examples (attached) which were created by the IRA Office and Shared with the Institutional Effectiveness Committee. The rubric is designed to reflect the expectations held by SACS reviewers for good academic and non-academic program assessment . Units will be judged  to comply or not comply with standards for correct assessment. Any Additional pertinent criteria not specifically addressed in the rubric or scoring examples  will be agreed upon by the SACS Liaison, Members of the IE Committee and the IRA Office and communicated orally and in notes to the units being coached.)   

The IRA office will promote and assist units to achieve 100% compliance across the university with regard to the SACS reaffirmation process.

     Compliance criteria are specified in the indicator rubric and scoring examples. Key characteristics of good assessment that are required for an OAT DB entry to be judged compliant include
          -Goals are broad and visionary.
          -Objectives for academic units must state learning outcomes that can be measured and that are non-trivial.
          -Objectives for non-academic units may be performance outcomes that can be measured and that are more important than outcomes required of routine operational tasks.
          -Indicators describe what will be measured, or checked to ascertain if the stated objective is or is not achieved.
          -Criteria describe the specific benchmarks that the indicator measurement will meet or exceed.
          -Findings will describe the results obtained from measuring the outcome for the objective and will discuss their interpretation in relation to meeting, exceeding or not meeting the criteria.
          -Actions will describe what the unit did with and as a result of the assessment findings for the purpose of improving the unit in relation to the attainment of the related objective or other related aspects of the program.
          -Closing the loops statement - summarizes improvements that may be under consideration, planned, or in progress  for the unit, currently or in the future,  as a result of what was learned from the assessment for that year.

Results:
By the first week of July 2010 the IRA Office delivered substantive accreditation related assessment planning and reporting assistance to 46 units through individual coaching with OAT DB managers and their delegated OAT DB reporter; through group demonstrations and presentation; and through extended written and phone exchanges of annotated OAT DB entries,instructions and step by step guided processes. At least six  of the 47 (RECALCULATE WITH ALL LEVELS COUNTED) units assisted were  academic units

By early July of 2010,  97 or 75% of the 130 units assisted, were judged to have Assessment Reporting in the OAT DB that were  at least minimally compliant with the expectations held by SACS reviewers for good academic and non academic assessment. However, 33 or 25% were considered to be fully compliant by the IRA review team.

Members of the IRA Assessment team have more subtle concerns and thoughts about the status of Assessment than those that were targeted for the Objective as stated in this reporting.

 The Assessment team of the IRA Office invested and expended a great deal time and energy in answering questions, delivering advice and modeling good assessment practices throughout the year, yet when the final reviews of annual assessment documentation were made, a number of the units substantively assisted in past years and some assisted in the current year, displayed evidence of not understanding assessment principles and definitions which had been demonstrated on prior occasions.  This loss of ground gained is disturbing to us and we have been considering reliable ways of gathering information about why this occurs and where the assistance that we deliver lacks the ability to make learning stick.

 The follow-up coaches appointed by the SACS Liaison have provided smatterings of critical information about the status of assessment competence displayed by units that IRA has served in the past, to whom they have delivered additional coaching in spring and summer of 2010. This feedback may not be strictly objective and unbiased, but it does serve as a point of departure for an investigation into the reasons for the lack of long-term or midterm effectiveness of the learning facilitation conducted by the IRA Office.

 Cultivating full acceptance of and knowledge about correct practices of formalized self assessment is a difficult and long-term process across any organization. Obtaining truly informative and reliable data to use in designing quality improvements in the long-term effectiveness of IRA Assessment coaching would benefit the university and the professional satisfaction of IRA Assessment Team members.

Promote responsive, welcoming and customer oriented service

SHSU decision makers will be able to use and understand the data they are provided by the IRA Office

The IRA Office will use the Workorder Request Table to track the number of repeat clients in the 2009-2010 year as a comparison to the numbers gathered in Fiscal Year 2008 and Fiscal Year 2009. The fact the clients are making multiple data requests of our office is an indication that they are not only generally pleased with the quality of the work being done, but that they are able to both understand and use the information being given to them. In addition, analysis of the information recorded in the Workorder Table will be conducted to determine if the the IRA Office is serving clients from across the University.

The IRA office would like to see an increase in the number of repeat clients in 2009-2010.

At this time, we may only attempt to draw a conclusion about job usefulness by inference : If a client is a repeat user of IRA service, we surmise that he or she has used the IRA job results for the purpose intended and is hence returning to have such needs addressed again. 

From the 2009-2010 IRA Work Order table, there were 57 repeat clients within that year, which amounts to a decrease of 4 from the previous year. However, this same year saw 61 repeat clients from the 2008-2009 Work Order table, 45 of which submitted more than one workorder in FY10. This is an increase from FY09 where 49 clients repeated from the previous year, 40 of those submitting more than one workorder in FY09.

The Office of Institutional Research and Assessment will conduct a year-end "Client Satisfaction Survey" of its clients, both internal and external, from that year. This survey will include items addressing satisfaction with accuracy, timeliness, and usefulness of data provided by the IRA Office, as well as client perceptions of the office's efforts regarding assistance with unit assessment. This survey was created by the staff of the IRA Office, and a meeting is held yearly to make sure that the survey is up-to-date.

A minimum 75% of survey respondents will not only be satisfied with the IRA office in general, but also with the accuracy, timeliness, and usefulness of the data they receive as a result of their work orders/data requests.

The 2009-2010 IRA Client Satisfaction Survey was sent to 81 former clients, from which was received 21 responses.  This equates to a response rate of 25.92%.  
     Customers appear to be satisfied with the usefulness, clarity and thoroughness of information reported to them as a result of research  jobs submitted. However, whether the clients are making use of the data can only be extrapolated from these survey findings.  This question may need to be addressed in the next survey. 
     At this time, we may only attempt to draw a conclusion about job usefulness by inference : If a client is a repeat user of IRA service, we surmise that he or she has used the IRA job results for the purpose intended and is hence returning to have such needs addressed again. 
This measurement and its results are discussed and reported in a separate Indicator and Finding for this Objective.
 
DETAIL SUMMARY OF SURVEY RESPONSES:
The responses from the participants who were able to form an opinion are summarized as follows:
95.2% were satisfied with the time it took to receive their results.

100% were satisfied with the presentation style or format of the information they received.

100% were satisfied with the customer service they received from the IRA Staff.

100% were satisfied with the usefulness of the information they received.

100% were satisfied with the clarity of the information they received.

100% were satisfied with the thoroughness of the information they received. 

After reviewing this data, overall, the IRA Office concluded that even though it has continued to meet or exceed its general requirements with regard to client satisfaction, a 25% response rate obtained possibly months after particular jobs are completed is not the most authentic basis for evaluating true, realtime satisfaction of clients.

Actions for improvement that have been undertaken as a result of the FY10 assessment for this and other objectives are described below:

    It is evident that the IRA Office would benefit from finding a way to obtain direct information about the usefulness which IRA customers ascribe to the services and products they receive from IRA.  This is currently being inferred from repeat use or IRA services by its customers. While this may be valid behavioral evidence, richer information about the nature of the uses to which IRA data are being put would be illuminating and would permit assessment of our performance variation re. the type of use for which our jobs are applied. Other benefits would result from having this information, as well. It could be easier to anticipate jobs in relation to needs that were tied to particular cycles, and needs that related several different jobs.

     Review of the survey findings overall lead to the conclusion that a 25% customer response rate, obtained possibly months after particular jobs are completed, is not the most authentic basis for evaluating true, real-time satisfaction of clients. Consequently, the IRA Office has concluded that we should attempt to revise our survey to include a more direct question about usefulness of IRA Jobs and the particular uses to which it was put, in addition to changing the collection process to obtain responses to IRA jobs in a real-time manner. This could improve the freshness of the resulting data and also the response rate.

    To this end, the IRA office staff once again (5th time in as many years) discussed desirable solutions and actionable options for actualizing them, and then called in technical consultation from Computer services. We again discussed, with CIS, the possibility of (1)changing the web based IRA WO form to collect more data about jobs directly from clients when they submit their requests, as well as the possibility of linking this data directly to a database. This would permit us to electronically and efficiently perform the tasks which we conducted manually to arrive at the results of this assessment. (2) Generating automatic email contact with clients when jobs are completed and delivered to direct clients to an online survey for satisfaction feedback about their job.

     While it appears that the answer to our requests will probably again be negative,  we are exploring other options which might yield a compromise solution for improving our collection of information about submitted jobs and our surveying of clients immediately after job deliveries.

Federal, State, TSU System, and SHSU administrative and academic consumers of SHSU data will be satisfied with the information and services provided to them by the IRA Office in a number of areas, including Accuracy, Timeliness, Consistency.

The Office of Institutional Research and Assessment will conduct a year-end "Client Satisfaction Survey" of its clients, both internal and external, from that year. This survey will include items addressing satisfaction with accuracy, timeliness, and usefulness of data provided by the IRA Office, as well as client perceptions of the office's efforts regarding assistance with unit assessment. This survey was created by the staff of the IRA Office, and a meeting is held yearly to make sure that the survey is up-to-date.

A minimum 75% of survey respondents will not only be satisfied with the IRA office in general, but also with the accuracy, timeliness, and usefulness of the data they receive as a result of their work orders/data requests.

The 2009-2010 IRA Client Satisfaction Survey was sent to 81 former clients, from which was received 21 responses.  This equates to a response rate of 25.92%.  
     Customers appear to be satisfied with the usefulness, clarity and thoroughness of information reported to them as a result of research  jobs submitted. However, whether the clients are making use of the data can only be extrapolated from these survey findings.  This question may need to be addressed in the next survey. 
     At this time, we may only attempt to draw a conclusion about job usefulness by inference : If a client is a repeat user of IRA service, we surmise that he or she has used the IRA job results for the purpose intended and is hence returning to have such needs addressed again. 
This measurement and its results are discussed and reported in a separate Indicator and Finding for this Objective.
 
DETAIL SUMMARY OF SURVEY RESPONSES:
The responses from the participants who were able to form an opinion are summarized as follows:
95.2% were satisfied with the time it took to receive their results.

100% were satisfied with the presentation style or format of the information they received.

100% were satisfied with the customer service they received from the IRA Staff.

100% were satisfied with the usefulness of the information they received.

100% were satisfied with the clarity of the information they received.

100% were satisfied with the thoroughness of the information they received. 

After reviewing this data, overall, the IRA Office concluded that even though it has continued to meet or exceed its general requirements with regard to client satisfaction, a 25% response rate obtained possibly months after particular jobs are completed is not the most authentic basis for evaluating true, realtime satisfaction of clients.

Findings indicated that relative to other responses, customers appear to be least satisfied with the time it takes for IRA to conduct analyses and generate reports of the results. The IRA Office is very quick to implement studies, generate and deliver reports that pertain to Mandates (internal or external) and these jobs are permitted to derail jobs in progress or in the Queue.  Because staff will not be increasing in size, strategic prioritization of jobs will continue to be necessary. However it may be very helpful for IRA to regularly send an explanatory group email to clients whose work has been bumped by a job identified as mandatory and priority with immediate or short deadline. 

     Dissatisfaction with the time involved to complete a request may also be a function of customer expectations mismatched to the actual complexity of a request. A policy practice is under discussion among the staff about sending a second contact to a client submitting a Work Order (first contact is generated automatically when the WO is submitted through he IRA Web Page) in which the director, Sr. Analyst or the Analyst I summarizes her understanding of the request along with her understanding of what will be entailed to produce it and the approximate time under the conditions of the moment. ( i.e., ., Is the only one analyst available for the week due to illness or due to high priority job that requires all hands to complete by short deadline). This communication is currently a normal course of business for priority jobs and jobs that require clarification; however it is not a blanket practice. As a blanket practice, this additional step will usurp analyst resources from analytical and reporting work; however, communication to influence perception may be more important than producing work as quickly as humanly and technically possible.

     After reviewing findings overall, the IRA Office members concluded that even though it has continued to meet or exceed its client satisfaction targets,  a 25% customer response rate, obtained possibly months after particular jobs are completed, is not the most authentic basis for evaluating true, real-time satisfaction of clients. Consequently, the IRA Office has concluded that, it is dissatisfied with the manner in which this Client Satisfaction Data is being collected, and that the Office should attempt to collect this data in more of a real-time manner in order to obtain a higher response rate, and fresher, better quality data.

     Consequently, for possibly the 5th time in the past 5 years, the IRA office staff called in technical consultation from Computer Services in order to again discuss the possibility of changing the IRA WO table to collect data directly to a WO data base and to automatically generate a contact with customers inviting them to complete a satisfaction survey for their recent job.

     There have always been insurmountable but reasonable technical, policy or resource barriers presented by CIS to accomplishing this proposal, however, IRA access to CIS permissions and direct access to a CIS employee dedicated to our data issues has brought us closer to being able to come to some compromise solution in the coming year.

Actively pursue approaches to recruit and retain quality staff.

Fill all open staff positions.

The IRA office will recruit individuals to occupy the positions of Senior Analyst, Analyst I, and Assessment Staff Assistant.

Fill all open positions with very competent personnel, while remaining within the prevailing constraints of salary budgets.

Senior Analyst position filled by exceptionally competent individual, who was promoted from the full time Analyst I position.  Analyst I position filled through promotion of previously employed IRA Graduate Assistant.  Assessment Staff Assistant position filled, and funded from allocations made from Banner ERP staffing backup.

Pursue the funding to create a permanent staff assistant position for assessment.  Formulate a plan for developing a new inexperienced entry level hire with customized competencies for the position in order to increase the likelihood of filling a new Staff Assistant position at the low salary level which will probably be allocated.  Develop alternative funding for an additional IRA graduate assistant to fill some of the gap that will remain when and if the current Staff Assistant position ends, if it is not renewed.