Special Education MA/MED (Low Incidence Disabilities And Autism)
Goal
Mastery Of Behavior Analysis Certification Board Task List Knowledge And Skills
Mastery of Knowledge and Skills based on Behavior Analysis Certification Board Task List.
Objective
Comprehensive Exams
Candidates will demonstrate mastery of knowledge and skills on comprehensive examinations, referenced to the standards set forth by the Behavior Analysis Certification Board.
Indicator
Scoring Rubric
Faculty-developed rubric scored by two independent faculty members following standards of the Behavior Analysis Certification Board Task List.
Criterion
Passing Rates On Comprehensive Examinations
90% of candidates will score 80% or better on the rubric (see attached). Particular areas of emphasis include data analysis, experimental design, behavioral support plan development, and application of ethical principles.
Because 70% of candidates scored above 80% on their initial attempt in 2012-2013, we will maintain the criterion of 90% of candidates. The domain identified to improve is the area of research design. We have set in motion some interventions for this area.
Curriculum changes are being made to focus more heavily on the task list created by the Behavior Analysis Certification Board. Furthermore, all classes in the future will be taught by Board Certified Behavior Analysts or individuals with a strong background in behavior analysis only.
In addition, we will be transitioning to written comprehensive exams in order to more fully capture the extent of a candidate's knowledge, and not their nerves at having to perform in person. Thus, this year, we will give candidates a choice of oral or written examinations as we make the transition. Therefore, our scores may be higher than usual for this reason in the following year.
Finding
Comprehensive Exam Findings.
Fifty percent of the students scored 80% (40 out of 50) or better upon the initial testing. However, because two of the students taking the exam orally were within 10% of the passing criteria (5 points), the faculty brought the students back into the room and probed for additional information on questions upon which they had performed poorly. Therefore, with additional prompting, those students achieved a passing grade (see the attached score sheet for initial and final scores).
Four students completed the exam in a written format. Of those, one passed, one was within the range to qualify for additional questions, and two failed. The one student who earned within a 35 initially passed when given two more questions orally. The two students who failed initially passed when presented with the test orally. Thus, we will be utilizing only oral exams in the future.
The average of all initial scores was 38.09 and the average of all final scores was 42.25 out of 50. Interrater reliability was very high across all students with an overall average of 89.75 and a range of 66.67-100 on individual questions.
Action
Comprehensive Examinations In LIDA
Cirriculum changes are being made to focus more heavily on the task list created by the Behavior Analysis Certification Board.
Furthermore, all classes in the future will be taught by Board Certified Behavior Analysts or individuals with a strong background in behavior analysis only.
Because only 50% of students met the goal on the first testing, this criterion will be maintained.
Objective
Formative Evaluation Of Behavior Analytic Terms (FEBAT)
Candidates will demonstrate mastery of knowledge and skills on the Formative Evaluation of Behavior Analytic Terms (FEBAT), referenced to the standards set forth by the Behavior Analysis Certification Board.
Indicator
Formative Evaluation Of Behavior Analytic Terms (FEBAT)
The FEBAT is a formative evaluation that assesses knowledge on 25 standard behavior analytic terms at six points in the candidate's academic career (see attached- beginning of program, end of first semester, end of second semester, end of summer term, end of first semester of second year, and end of program). Candidates receive course points for completion of the FEBAT, regardless of accuracy on the individual items. Therefore, the only environmental variables that evoke and maintain correct responding are instructional control in the classroom and a history of reinforcement for performing accurately on similar tasks. In addition, the students are urged to do well to provide an accurate assessment of their knowledge with respect to the material as they advance throughout the program.
The FEBAT is scored on a faculty-created rubric with a 5-point scale. Scores from 0-4 are administered based on the quality of the definition provided for each term; 4-point answers are those most similar to the definitions provided by Cooper, Heron, and Heward (2007), a textbook based on the Behavior Analysis Certification Board task list. At least 25% of FEBAT administrations are scored by two Board Certified Behavior Analysts to ensure interrater reliability.
Criterion
Passing Rates On FEBAT
80% of candidates will score 80% or better on the FEBAT rubric (see attached) at the final administration. Because this is a new objective for the 2013-2014 school year, we are setting a moderate initial criterion.
Curriculum changes are being made to focus more heavily on the task list created by the Behavior Analysis Certification Board. Furthermore, all classes in the future will be taught by Board Certified Behavior Analysts or individuals with a strong background in behavior analysis only. Therefore, achievement of the criterion in the 2013-2014 school year seems plausible.
Finding
Passing Rates On FEBAT
No student exceeded 80% on the final administration of the FEBAT. The average score was 50.55 with a range of 35 to 65.5. Due to failure to meet this goal, this criterion will be maintained. See attached for individual scores.
Criterion
Demonstrated Improvement On The FEBAT
Because the FEBAT is administered six times (five times for the 2013-2014 cohort, see attached list) during a candidate's graduate career, it serves as a method of formative evaluation. Therefore, students will improve their performance from the initial administration of the FEBAT to the final administration by 20%.
Finding
Demonstrated Improvement On The FEBAT
Eight out of nine students improved their scores on the FEBAT by at least 20% from the initial administration (average 31.2%). However, given the overall low scores, we will maintain this criterion next year.
Action
Formative Evaluation Of Behavior Analytic Terms (FEBAT)
Cirriculum changes are being made to focus more heavily on the task list created by the Behavior Analysis Certification Board. Furthermore, all classes in the future will be taught by Board Certified Behavior Analysts or individuals with a strong background in behavior analysis only. Additionally, review cards featuring terms similar to those in the FEBAT have been incorporated into classes.
Objective
Behavior Analysis Certification Exam
Candidates will pass the Behavior Analysis Certification Exam at percentages comparable to the national average.
Indicator
Percentage Of Candidates Passing Behavior Analysis Certification Exam
The Special Education MA/MED (Low Incidence Disabilities And Autism) graduate program prepares students to take the Behavior Analysis Certification Exam, the final step in becoming a Board Certified Behavior Analyst. Therefore, the true test of the program's effectiveness is the percentage of students passing the exam every year.
The Behavior Analyst Certification Board reports the percentage of students passing the exam and the national average. In 2012, a total of 9 first-time test takers completed the BCBA; 3 passed the examination, resulting in a pass-rate of 33%. The reported national data for 2012 showed a passing rate of 58%; thus, our passing rate was significantly below national average.
Criterion
Passing Rate On BCBA Exam
As a measure of continuous improvement, the criterion for this objective is that the passing rate for each year exceed the passing rate for the previous year as it approaches the national standard. Annual data (starting in 2012) will be graphed and progress will be tracked as the program continues to expand and develop. Therefore, the passing rate for 2013 must exceed 33%. Note that data are available each academic year for the candidates who graduated the previous year.
Finding
Passing Rate On BCBA Exam
The Behavior Analyst Certification Board reports the percentage of students passing the exam and the national average. In 2013, a total of 11 first-time test takers completed the BCBA; 4 passed the examination, resulting in a pass-rate of 36%. The reported national data for 2012 showed a passing rate of 58%; thus, our passing rate was significantly below national average.
Action
Passing Rate On BCBA Exam
As a measure of continuous improvement, the criterion for this objective is that the passing rate for each year exceed the passing rate for the previous year as it approaches the national standard. Annual data (starting in 2012) will be graphed and progress will be tracked as the program continues to expand and develop. Therefore, the passing rate for 2014 must exceed 36%. Note that data are available each academic year for the candidates who graduated the previous year.
Goal
Production Of Scholarly Research Project
Candidates will write a research paper, conforming to APA style
Objective
Research Proposal
Candidates will develop a high-quality research proposal in SPED 6314. Candidates will demonstrate mastery of this skill by the end of 6317.
Indicator
Faculty Developed Rubric
The research proposal will be scored by instructors of SPED 6314.
Criterion
Passing Rate On The Rubric
To evaluate the research proposal in SPED 6314, 80% of the candidates will score 80% or better on the attached rubric. Please note that the rubric has changed from last year to more closely reflect the sections, subsections, and writing style of research manuscripts in the field of behavior analysis.
Because only 60% of students achieved the 80% criterion last year, it will be maintained this year. Last year, the area identified to improve was written description of procedure on the research proposal. This year, we will maintain that goal in addition to targeting written description of results.
Finding
Research Proposal In SPED 6314
Candidates submit a research proposal at the end of the semester. The proposals are scored with the attached rubric that looks at competency domains such as "Introduction", "Method," and "Description of Procedure." Students' performance (n = 10) on the 160 point rubric averaged 82.9%. Range of performance was 61.6-95.7%. The lowest domain of candidate performance was in the area of results. Candidates performed well on providing background information on subject selection, identification of target skill, etc.
Findings from this year show a 5.2% average improvement from last year.
Criterion
Improvement Of Research Project From Fall (SPED 6314) To Spring (SPED 6317) Of Last Year
We have added a second criterion in 2013-2014 to evaluate continuing improvement on this goal in future years. Candidates complete final drafts of their research projects in both SPED 6314 (Fall) and SPED 6317 (Spring). These are evaluated using the same rubric, the updated rubric used to also assess the first criterion (see above). Therefore, we would like to see a 10% increase in scores on 80% of candidates' final project in SPED 6317, relative to the score they received on the final project they submitted in SPED 6314. The range for improvement will vary based on the candidates' initial performance, but a 10% increase for 80% of candidates is feasible, given the low percentage of candidates that score above 90% on the project in SPED 6314. To achieve this goal, we intend to have the candidates submit multiple drafts throughout the semester to allow us to shape their written behavior.
Finding
Improvement Of Research Project From Fall (SPED 6314) To Spring (SPED 6317) Of Last Year
To determine improvement in scientific writing ability, scores were compared on the student's research projects from Fall to Spring semesters. All students improved across semesters; only six out of eleven improved more than 10% relative to their Fall performance. The average scores out of 160 points were 130.1 (81.3%) and 151.7 points (94.8%) in Fall and Spring, respectively. Furthermore, the average improvement from Fall to Spring was 18.3% when calulated as percentage of Fall scores and 13.5% when calculated as percentage of the total 160 points. All students made noticeable improvement in their writing across semesters, even if they did not meet the criterion.
Action
Research Proposal In SPED 6314
Students will begin working on their research proposals much earlier in their academic careers (Spring semester of their first year).
Students will be introduced to a wider variety of research throughout their classes.
Students will receive increased emphasis on written description of procedures - clarity and conciseness - in the early portion of the course. Students will be afforded opportunities to submit drafts of the proposal prior to the final submission at the end of the course.
The accuracy criterion of 80% of candidates was not obtained - two candidates fell short in their initial drafts in SPED 6314. Thus, 80% will remain the criterion level of accuracy for the research proposal.