Home / RBT Study Guide / Data Collection and Graphing / Data reliability, procedural fidelity and summarizing data
Domain A · Data Collection and Graphing · Tasks A.3, A.6, A.8
Data reliability, procedural fidelity and summarizing data
Reliable data means the numbers match what actually happened. Procedural fidelity means the plan was run as written. When either one slips, the supervisor ends up making treatment decisions from a graph that is not true.
Last reviewed November 10, 2026 against the BACB RBT Test Content Outline (3rd ed.)
Why unreliable data is a risk
Task A.8 of the RBT Test Content Outline asks you to describe the risks of unreliable data collection and poor procedural fidelity. The core risk is simple: your supervisor makes treatment decisions from the graph. If the numbers are wrong, the decisions can be wrong.
- An effective plan may be changed or dropped because the data hide the progress.
- An ineffective plan may continue because the data look better than reality.
- A client may be moved to a harder target before a skill is solid, or kept on a mastered one.
- Time, and the family's trust, are lost.
Data usually become unreliable for ordinary reasons: recording from memory after the session, a vague definition, guessing at interval timing, or being too busy to tally.
Procedural fidelity
Procedural fidelity (also called treatment integrity) is how closely the plan is implemented as written. Giving a prompt the plan does not call for, skipping reinforcement, or giving attention when the plan says to withhold it are all fidelity errors.
Poor fidelity has its own risks. If the plan was not run as written, no one can say whether the plan works. Inconsistent implementation can also slow learning or accidentally reinforce the behavior the plan targets.
In session
Leo's plan says to deliver a token after every correct response. His RBT, juggling materials, delivers tokens about half the time. Leo's accuracy stalls. If the supervisor does not know about the missed tokens, she may decide the token system does not work for Leo, when it was never delivered as designed. The RBT's job is to say so.
Interobserver agreement, drift and reactivity
The usual check on data is interobserver agreement (IOA): two people record the same behavior, in the same session, independently, and then compare.
- Total count: smaller count divided by larger count, times 100. Counts of 18 and 20 give 18 / 20 = 0.90, or 90%.
- Interval by interval: intervals agreed divided by total intervals, times 100. Agreement on 17 of 20 intervals gives 17 / 20 = 0.85, or 85%.
A common convention treats 80% or higher as acceptable, but your supervisor sets the standard for each program. Low agreement does not show who is wrong. It shows the definition or the recording needs attention.
Two threats come up often:
- Observer drift. Over time, an observer unintentionally shifts how they apply the definition, counting things that used to be excluded or the reverse.
- Reactivity. Behavior changes because someone is watching. A client may act differently with a new observer in the room, and staff often record more carefully when a supervisor is present.
What you do about it:
- Re-read the operational definition regularly and follow it as written.
- Record during the session, as behavior happens, not from memory.
- Tell your supervisor promptly about missed data, a fidelity error or a definition that is hard to apply. Never fill in or adjust numbers to look complete.
Common mix-up
Drift is a slow change in the observer's scoring. Reactivity is a change in behavior caused by being observed. If the scenario says "gradually" or "over several weeks," think drift. If it says "when the supervisor was in the room," think reactivity.
Permanent product recording
Task A.3 covers permanent product recording: measuring a behavior by the lasting result it leaves, after the behavior is over. You count the product, not the action. Examples include worksheet problems completed, toys in the bin, beads strung, or words typed.
The advantage is that you do not have to watch the behavior, so you can teach during the work period and the product can be checked later by a second person. The limits are just as important. The behavior has to leave a consistent product, and you need to be confident the client produced it. A finished worksheet does not tell you whether a peer helped or how long it took.
Calculating and summarizing data
Task A.6 names three ways to summarize raw data. Each turns a session's tallies or times into one number that can go on the graph.
| Summary | Formula | Worked example |
|---|---|---|
| Rate | Count divided by observation time | 12 hits in 30 minutes: 12 / 30 = 0.4 per minute, or 24 per hour |
| Mean duration | Total duration divided by number of occurrences | Tantrums of 3, 5 and 10 minutes: 18 / 3 = 6 minutes per tantrum |
| Percentage | Part divided by whole, times 100 | 14 correct of 20 trials: 14 / 20 = 0.70, or 70% |
Use rate when session lengths differ, because counts from a 30-minute and a 60-minute session cannot be compared directly. Use mean duration when the length of each episode matters. Use percentage for trial-based teaching and for interval data, where the result is the percentage of intervals scored. The measures behind these numbers are explained under continuous measurement.
On the exam
For calculation items, write the formula before touching the numbers, and check the units. Rate answers carry a time unit (per minute, per hour). A mean duration divides by the number of episodes, not by session length. A percentage can never be above 100, so an IOA answer over 100 means the division was upside down.
When data or fidelity problems happen, how and when you tell your supervisor falls under documentation and reporting.
Practice questions
Original questions in the style of the exam. Pick an answer before you open the explanation.
1. Hitting is defined as forceful contact with an open or closed hand. Over several months, Priya gradually starts counting light taps as hits too. What is this an example of?
- Observer reactivity
- Observer drift
- Treatment drift
- Observer expectancy
Show answer
B. Observer drift
Observer drift is a gradual, unintended shift in how an observer applies a definition. Reactivity is a change in behavior caused by being observed, and treatment drift is about how the intervention is delivered, not how the behavior is scored.
2. Two observers independently count Sam's requests during the same session. One records 16 and the other records 20. What is the total count interobserver agreement?
- 64 percent
- 90 percent
- 125 percent
- 80 percent
Show answer
D. 80 percent
Total count agreement is the smaller count divided by the larger count, times 100: 16 / 20 = 0.80, or 80 percent. Dividing the larger by the smaller gives 125 percent, which cannot be right because agreement never exceeds 100 percent.
3. After a session, an RBT counts how many math problems Mia completed on her worksheet without having watched her work. Which procedure is this?
- Permanent product recording
- Momentary time sampling
- Whole interval recording
- Continuous event recording
Show answer
A. Permanent product recording
Permanent product recording measures the lasting result of a behavior after it has occurred. Continuous event recording would mean tallying each response as it happened, which requires watching the behavior live.
Common questions
What is procedural fidelity in ABA?
Procedural fidelity, also called treatment integrity, is the extent to which a plan is carried out exactly as written. When fidelity is poor, the team cannot tell whether the plan itself works, and the client may make less progress or have problem behavior reinforced by accident.
How do you calculate interobserver agreement (IOA)?
For a simple total count check, divide the smaller count by the larger count and multiply by 100. If one observer counts 18 and the other counts 20, IOA is 18 / 20 x 100 = 90 percent. For interval data, divide the number of intervals where both observers agreed by the total number of intervals and multiply by 100.
What is permanent product recording?
It is measuring a behavior by the lasting result it leaves, such as completed worksheet problems, dishes put away or items assembled. The observer does not need to see the behavior happen. It only fits behaviors that reliably leave a product that can be tied to the client.