Procedural Integrity Matters
Treatment integrity data is critical to capture during supervisory observations. Simply, treatment integrity is the degree to which an intervention is implemented as designed (i.e., quality; Sanetti & Kratchowill, 2009). Not only should treatment integrity be collected to guide supervisory practice in terms of determining how frequently to conduct observations. It is also important to collect because it impacts clinical outcomes (e.g., DiGennaro Reed et al., 2007). Additionally, the BACB requires that supervisors collect data on clinician performance as part of the BCBA code of ethics. If a client is not making progress, and supervisors do not have integrity data on clinician performance, how will they know why progress has stalled? Is it that the clinician is not performing as the plan was designed? Is it that modifications need to be made to the client's protocols?
Guide Supervisory Practices
When observations occur, supervisors should take data on treatment integrity. Our last installment discussed that supervisors must develop tools (e.g., integrity checklists) to support their observation efforts. To create the checklists, supervisors should task analyze the behavior they would like to measure. The number of checklists created should be based on what the supervisor wants to observe. All of the items do not need to be included in one, long assessment; rather, smaller assessments across implementation categories (e.g., behavior reduction, skill acquisition) could be created to guide supervisory observations. The data gathered using the tools developed should then be analyzed to guide a supervisor’s subsequent action. I am often asked how frequently a clinician should be supervised. My answer is: let the data guide your behavior! Here are some questions that a supervisor can ask themself: What is the integrity level for a particular clinician? Are the data revealing areas for improvement? How is my team performing in the area of behavior reduction? Do additional training or interventions need to be implemented to improve performance?
Impacts Clinical Outcomes
Treatment integrity is critical because research has shown that it is correlated with clinical outcomes. Research has evaluated what level of integrity is necessary for clinical progress to occur. Currently, there is no specific level of integrity that is the gold standard. Higher levels of integrity more reliably reveal clinical progress than lower levels of integrity. Dr. St. Peter’s research has revealed that participants still made progress when integrity was at 80%. A recent study in JABA by Dr. Denys Brand’s lab found little difference in client outcomes when integrity was 80 or 100%. However, when initial integrity errors (i.e., integrity was 20%) occurred clients struggled to make progress when integrity reached 80%. Additionally, multiple types of errors are detrimental to learning. These data suggest that it is critical that all clinicians implement protocols with high levels of integrity from the onset of their service delivery!
Supervision Requirement
As noted above, the BACB requires that supervisors collect and use integrity data as part of the supervisor's responsibility to the clinician. The number of clinicians, and BCBAs, in our field continues to grow at a rapid rate. It is the responsibility of the supervisor to ensure their clinicians have the skills necessary to effectively work with clients. Without effective supervisory oversight and support, clinicians are likely to continue to lack the skills necessary to evoke behavior change. Work with your clinicians to make supervision a fun, rewarding process where the goal is to improve the skill set of the clinician and deliver the best services possible to the clients.
Additional Information on Treatment Integrity
There are other forms of treatment integrity that are discussed in the literature. Data could also be collected on content, quantity, and process.
Content: what steps were delivered accurately? These data provide supervisors with information about what specific behaviors/steps to a protocol that clinicians need additional support with.
Quantity: how much of the intervention was provided? These data provide supervisions with the information to adjust the dosage of the intervention, if needed.
Process: how was the intervention delivered? These data provide supervisors with information about whether the process of service delivery needs to be modified.
References
Behavior Analyst Certification Board (2020). BCBA code of ethics. Retrieved from: https://www.bacb.com/wp-content/uploads/2022/01/Ethics-Code-for-Behavior-Analysts-220316-2.pdf
Brand, D., Henley, A. J., DiGennaro Reed, F. D., Gray, E., & Crabbs, B. (2019). A Review of Published Studies Involving Parametric Manipulations of Treatment Integrity. Journal of Behavioral Education, 28, 1–26. https://doi.org/10.1007/s10864-018-09311-8
DiGennaro, F. D., Martens, B. K., & Kleinmann, A. E. (2007). A comparison of performance feedback procedures on teachers’ treatment implementation integrity and students’ inappropriate behavior in special education classrooms. Journal of Applied Behavior Analysis, 40, 447–461.
Falakfarsa, G., Brand, D., Bensemann, J., & Jones, L. (2023). A parametric analysis of procedural fidelity errors following mastery of a task: A translational study. Journal of Applied Behavior Analysis, online first.
Sanetti, L. M. H., & Kratchowill, T. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38(4), 445-459.
St. Peter Pipkin, C., Vollmer, T. R., Sloman, K. N., & Roane, H. S. (2010). Effects of treatment integrity failures during differential reinforcement of alternative behavior: A translational model. Journal of Applied Behavior Analysis, 43(1), 47–70. https://doi.org/10.1901/jaba.2010.43-47