Special Education MA/MED (Low Incidence Disabilities And Autism)
Goal
Mastery Of Behavior Analysis Certification Board Task List Knowledge And Skills
Mastery of Knowledge and Skills based on Behavior Analysis Certification Board Task List.
Objective
Comprehensive Exams
Candidates will demonstrate mastery of knowledge and skills on comprehensive examinations, referenced to the standards set forth by the Behavior Analysis Certification Board.
Indicator
Scoring Rubric
Faculty-developed rubric scored by two independent faculty members following standards of the Behavior Analysis Certification Board Task List.
Criterion
Passing Rates On Comprehensive Examinations
90% of candidates will score 80% or better on the rubric (see attached). Particular areas of emphasis include data analysis, experimental design, behavioral support plan development, and application of ethical principles.
Because 50% of candidates scored above 80% on their initial attempt in 2013-2014, we will maintain the criterion of 90% of candidates. The domains identified to improve is the areas of research design and verbal behavior. We have set in motion some interventions for these areas.
Curriculum changes are being made to focus more heavily on the task list created by the Behavior Analysis Certification Board. Furthermore, all courses with behavior analytic content will henceforth be taught by Board Certified Behavior Analysts or individuals with a strong background in behavior analysis only.
In the previous year, we had piloted written comprehensive exams in an effort to transition to this method. The students who took the exams via this method scored lower overall, and therefore, we will not be transitioning to written exams.
Finding
Comprehensive Exams Findings
Sixty six percent (10 of 15 ) of the students scored 80% (40 out of 50) or better upon the initial testing. However, because four of the students taking the exam orally were within 10% of the passing criteria (5 points), the faculty brought the students back into the room and probed for additional information on questions upon which they had performed poorly. Therefore, with additional prompting, those students achieved a passing grade (see the attached score sheet for initial and final scores).
One student did not score within 10% of passing on their initial attempt. The student was given feedback and was administered the exam one week later, during which time the student passed without additional prompting.
The average of all initial scores was 41.1 and the average of all final scores was 42.65 out of 50. Interrater reliability was very high across all students with an overall average of 95.6 and a range of 58.3-100 on individual questions.
Action
Comprehensive Exam Evaluations In LIDA
Curriculum changes are continuing to be made to focus more heavily on the Fourth Edition task list created by the Behavior Analysis Certification Board.
Furthermore, all classes in the future will be taught by Board Certified Behavior Analysts or individuals with a strong background in behavior analysis only. We also have hired a new LIDA faculty member.
Because only 66% of students met the goal on the first testing, this criterion will be maintained.
Objective
Behavior Analysis Certification Exam
Candidates will pass the Behavior Analysis Certification Exam at percentages comparable to the national average.
Indicator
Percentage Of Candidates Passing Behavior Analysis Certification Exam
The Special Education MA/MED (Low Incidence Disabilities And Autism) graduate program prepares students to take the Behavior Analysis Certification Exam, the final step in becoming a Board Certified Behavior Analyst. Therefore, the true test of the program's effectiveness is the percentage of students passing the exam every year.
The Behavior Analyst Certification Board reports the percentage of students passing the exam and the national average. In 2012, a total of 9 first-time test takers completed the BCBA; 3 passed the examination, resulting in a pass-rate of 33%. The reported national data for 2012 showed a passing rate of 58%; thus, our passing rate was significantly below national average.
Criterion
Passing Rate On BCBA Exam
As a measure of continuous improvement, the criterion for this objective is that the passing rate for each year exceed the passing rate for the previous year as it approaches the national standard. Annual data (starting in 2012) will be graphed and progress will be tracked as the program continues to expand and develop. Therefore, the passing rate for 2013 must exceed 33%. Note that data are available each academic year for the candidates who graduated the previous year.
Finding
Passing Rate On BCBA Exam
The Behavior Analyst Certification Board reports the percentage of students passing the exam and the national average. In 2014, a total of 10 first-time test takers completed the BCBA exam; 5 passed the examination, resulting in a pass-rate of 50%. The reported national data for 2014 showed a passing rate of 60%; thus, our passing rate was slightly below national average.
Action
Passing Rate On BCBA Exam
As a measure of continuous improvement, the criterion for this objective is that the passing rate for each year exceed the passing rate for the previous year as it approaches the national standard. Annual data (starting in 2012) will be graphed and progress will be tracked as the program continues to expand and develop. Therefore, the passing rate for 2015 must exceed 50%. Note that data are available each academic year for the candidates who graduated the previous year.
Objective
Formative Evaluation Of Behavior Analytic Terms (FEBAT)
Candidates will demonstrate mastery of knowledge and skills on the Formative Evaluation of Behavior Analytic Terms (FEBAT), referenced to the standards set forth by the Behavior Analysis Certification Board.
Indicator
Formative Evaluation Of Behavior Analytic Terms (FEBAT)
The FEBAT is a formative evaluation that assesses knowledge on 25 standard behavior analytic terms at six points in the candidate's academic career (see attached- beginning of program, end of first semester, end of second semester, end of summer term, end of first semester of second year, and end of program). Candidates receive course points for completion of the FEBAT, regardless of accuracy on the individual items. Therefore, the only environmental variables that evoke and maintain correct responding are instructional control in the classroom and a history of reinforcement for performing accurately on similar tasks. In addition, the students are urged to do well to provide an accurate assessment of their knowledge with respect to the material as they advance throughout the program.
The FEBAT is scored on a faculty-created rubric with a 5-point scale. Scores from 0-4 are administered based on the quality of the definition provided for each term; 4-point answers are those most similar to the definitions provided by Cooper, Heron, and Heward (2007), a textbook based on the Behavior Analysis Certification Board task list. At least 25% of FEBAT administrations are scored by two Board Certified Behavior Analysts to ensure interrater reliability.
Criterion
Demonstrated Improvement On The FEBAT
Because the FEBAT is administered six times (five times for the 2013-2014 cohort, see attached list) during a candidate's graduate career, it serves as a method of formative evaluation. Therefore, students will improve their performance from the initial administration of the FEBAT to the final administration by 20%.
Finding
Demonstrated Improvements On The FEBAT
All 15 students improved their scores on the FEBAT by at least 20% from the initial administration (average 53.3%). The large improvements in percentages were likely a function of low initial scores.
Criterion
Passing Rates On FEBAT
80% of candidates will score 80% or better on the FEBAT rubric (see attached) at the final administration. Because this is a new objective for the 2013-2014 school year, we are setting a moderate initial criterion.
Curriculum changes are being made to focus more heavily on the task list created by the Behavior Analysis Certification Board. Furthermore, all classes in the future will be taught by Board Certified Behavior Analysts or individuals with a strong background in behavior analysis only. Therefore, achievement of the criterion in the 2013-2014 school year seems plausible.
Finding
Passing Rates On The FEBAT
Despite the large improvements from the initial administration to the final administration, only 5 of the 15 students (33%) attained at least 80% correct on the final administration.
Action
Formative Evaluation Of Behavior Analytic Terms (FEBAT)
Curriculum changes are being made to focus more heavily on the task list created by the Behavior Analysis Certification Board. Furthermore, all classes in the future will be taught by Board Certified Behavior Analysts or individuals with a strong background in behavior analysis only. We will continue to provide review cards featuring terms similar to those in the FEBAT during classes.
Goal
Production Of Scholarly Research Project
Candidates will write a research paper, conforming to APA style
Objective
Research Proposal
Candidates will develop a high-quality research proposal in SPED 6314. Candidates will demonstrate mastery of this skill by the end of 6317.
Indicator
Faculty Developed Rubric
The research proposal will be scored by instructors of SPED 6314.
Criterion
Passing Rate On The Rubric
To evaluate the research proposal in SPED 6314, 80% of the candidates will score 80% or better on the attached rubric.
Because only 72.7% of students achieved the 80% criterion last year, it will be maintained this year. Last year, the area identified to improve was written description of procedure on the research proposal. This year, we will maintain that goal in addition to targeting written description of results.
Finding
Passing Rate On Research Proposal
Candidates submit a research proposal at the end of the semester. The proposals are scored with the attached rubric that looks at competency domains such as "Introduction", "Method," and "Description of Procedure." Students' performance (n = 15) on the 160 point rubric averaged 91.1%. Range of performance was 81.9-98.1%. The lowest domain of candidate performance was in the area of results. Candidates performed well on providing background information on subject selection, identification of target skill, etc.
Findings from this year show a 9.8% average improvement from last year.
Criterion
Improvement Of Research Project From Fall (SPED 6314) To Spring (SPED 6317) Of Last Year
It was observed in the previous year that a ceiling effect was occuring within our data. Only 6/11 students met the criteria of a 10% increase over baseline, but an additional 3 students were excluded who scored above a 95% on the rubric. Therefore, we are modifying the criterion to state that 80% of students will improve their initial score by 10% or score above a 95% on the final paper to meet criteria. To achieve this goal, we intend to have the candidates submit multiple drafts throughout the semester to allow us to shape their written behavior.
Finding
Improvement On Research Project From Fall To Spring
To determine improvement in scientific writing ability, scores were compared on the student's research projects from Fall to Spring Semesters. Five of the students showed improvements in scores, although small (average 2.11%). The average scores out of 160 points were 145.8 points (91.1%) and 141.5 points (88.4%) in Fall and Spring, respectively. Many of the students showed decreases in scores from Fall to Spring, but are likely a function of high initial scores. In addition, decreases may have been a result of additional data and analyses that were included in the final project and not the intial project.
Action
Research Project Improvement
Students will begin working on their research proposals much earlier in their academic careers (Spring semester of their first year).
Students will be introduced to a wider variety of research throughout their classes.
Students will receive increased emphasis on written description of procedures - clarity and conciseness - in the early portion of the course. Students will be afforded additional opportunities to submit drafts of the proposal prior to the final submission at the end of the course.
The accuracy criterion on the final submission of 80% of candidates was obtained, however scores decreased from fall to spring. Thus, 80% will remain the criterion level of accuracy for the research proposal.