To Help Students Succeed We Need to Collect the RIGHT Data

Why Haven’t They Done That Yet?

Proper Prior Preparation Prevents Piss Poor Performance

– US Army 7Ps

I will start this post with two fairly controversial assertions that I will defend later.

  1. In schools, we do not collect nearly enough data on student behavior
  2. When we do collect data, they are most likely the wrong data

You can take these points as the TL;DR of this post. We do not collect enough data on how students behave in school, and when we do we are more than likely to collect irrelevant data that do not help the student.

My solution is simple. In fact, spectacularly so: In education, we need to standardize our datasheets within the district, school, and team. I have a solution already available and am more than willing to work with anyone that wants to develop their own from scratch.

What are you talking about?

My illustrative example involves collecting data on every single instance of “out of seat” for a student I will call Robert rather than collecting data on the specific behaviors that disrupted the learning of others (e.g., touching other students).

The teacher was instructed to collect data on all “out of seats” for Robert. This was the full extent of instruction given to the teacher regarding the data collection. No data sheets were provided, no direct modeling, no behavioral topography, just the definition “out of seats“. The teacher repeated this instruction to the paraeducator.

During class, these data were almost always collected after instruction had ended and the data had to be recorded from “memory”, often on a lined paper on a clipboard that only had Robert’s name on the top and disorganized tic marks. The paraeducator, however, only recorded “out of seat” if it resulted in a classroom disruption as they determined that Robert moving up and down in their seat as a fidget rather than a behavior.

With those data, the interventionist programmed interventions designed to “fix” Robert’s getting out of their seat repeatedly. The plan did not, however, address the root problem of why Robert was touching his friends without permission. Based on the data, the interventionist also determined the teacher, but not the paraeducator, was a trigger for Robert’s behavior because the teacher had recorded a much greater quantity of “out of seats“.

I agree that if we keep Robert in his chair he will touch other students less, at least in the short term. But anyone who has experience with chronic friend-tappers knows that Robert will just start scooting his desk or looking for opportunities to be allowed to leave his seat and they increase the amplitude of the behavior when he has the chance (e.g., instead of gently touching he now smacks his friends in the head as he walks by or starts kicking his friends). Stopping “out of seat” behavior only served to make Robert’s behaviors worse!

At this point, I was called in and did what I describe in the Now for Solutions section to help restore order to the classroom and teach Robert how to behave in class.


In my experience watching students in class and watching those tasked with collecting data, I see a few trends that trouble me:

  • Teachers walk around with clipboards, but they do not collect the data as the behavior unfolds but rather they watch the behavior, wait for it to end, and only then they write down in a narrative form based on what they remember
  • Data are collected on lined paper by jotting down notes
  • Different members of the team collect different data on different students using different data sheets and then try to “compare notes”
  • Team members are not discussing the who, what, when, where, or why of data collection

 

Clipboards don’t mean data are collected in a timely manner

This is a matter of timing. When we are emotional or anxious, our memories of something that has happened can get corrupted or fade very rapidly. So rapidly, in fact, that if we wait as little as five minutes to document what happened we have difficulty remembering specifics and have to guess as to what we saw.

When we see a student misbehaving, I know the impulse is to intervene now and collect data and record it later. This is the wrong impulse unless you pass data collection to another person to take it on the fly. Optimally, we hand the clipboard to the nearest adult who is trained in taking data and they collect the data while we intervene. This is rarely the case. Teachers feel an obligation to be the one who collects all the data, does all the interventions, etc.

As described below, when we write our data we often add our emotions and interpretations into it, and we feel this is necessary as we are providing our judgment and expertise. Sadly, when we do this we are mistaken. We are contaminating our data.

When we take the data on the fly and rapidly, we often actually do a more accurate job because we do not have time to think, we only have time to focus on collecting data and observation. This is what we should strive for.

Data are collected on lined paper

This is not a bad idea, but unless a data sheet designed with the student and with the target behavior(s) in mind is attached to the clipboard, you might as well be scribbling notes on the back of your hand. My issue with anecdotal record keeping is it should only actually be done by trained professionals that know how to extemporaneously describe behavior in a dispassionate and unaffected manner.

Quick example:

Rhett ran up to Scarlet to scare her, angrily grabbed her face, wailed at her like a banshee, and intentionally yanked her hair as he bolted. When she yelled, he got angry and turned back to kick her as hard as he could in the shin. He then proudly laughed and ran away from the teachers.

Compare this to when I remove words that convey an interpretation

Rhett ran up to Scarlet in order to to scare her, angrily grabbed her face, wailed at her like a banshee, and intentionally yanked her hair as he bolted. When she yelled, he got angry and turned back to kick her as hard as he could in the shin. He then proudly laughed and ran away from the teachers.

Resulting in…

Rhett approached Scarlet, grabbed her face, yelled at her, and pulled her hair as he ran away. When Scarlet yelled, Rhett turned back, kicking her in the shin. He laughed, then ran to the other side of the classroom.

I primarily see the former in the anecdotal data that I receive. Especially when hair pulling, kicking, biting, or spitting are concerned. There are a lot of words describing how a behavior was done or why a behavior was done (motivation), and a dearth of information regarding the topography of the behavior or how the behavioral episode unfolded.

Words like: abrasive, abusive, angry, anxious, belligerent, boorish, cowardly, crazy, creepy, cruel, chucked, dangerous, defiant, erratic, finicky, flashy, flippant, foolish, furtive, guarded, intentionally, jittery, malicious, mysterious, obnoxious, outrageous, panicky, proudly, revenge, secretive, strange, threatening, unsuitable, vengeful, and wary get placed within descriptions of behavior (See a more comprehensive list here or here)

The reason I find the use of any interpretations laden with adjectives or adverbs troubling is that I can no longer trust the anecdotal data. Is the teacher watching what is happening or telling them self a story to explain the behavior and, in doing so, missing the critical subtle details? A good description of behavior involves only nouns, pronouns (though preferably not), and verbs. Nothing else is relevant. These type of data contain only the who, what, when, where, and how of a behavior. There are no “why” to the behavior at this point.

Different members of the team collect different data on different students using different data sheets

This is the part of the post where I start to rag on interventionists and behavior specialists that come in to work with or observe our students (and I do this as someone who has held that position so I am totally mocking myself as well). We in the district offices often demand that teachers seek out and use data sheets for their data collection; we then leave the teachers to their own devices to hunt on Teachers Pay Teachers or Google for available options.

The district personnel, however, walk in with a legal pad, sit down and write anecdotal notes of what a student is doing based on what they have prejudged as important. Then often they (we) make broad interpretations and sweeping conclusions based upon that cursory observation and incomplete data, often neglecting to debrief the teacher on the data collected (and we have just finished discussing the weakness of this approach).

My concern with this approach is that anecdotal sheets or lined paper do not define the behavior or how to collect the data. A lined paper will not help you determine if you want to collect frequency, latency, duration, the amplitude of behavior, etc. It is blank and unhelpful.

editable-primary-lined-paper
How on Earth do I collect data using this?

Even in the hands of a trained professional, lined paper is at best a useless, if not harmful, tool for data collection. I know I miss things when I have to write notes down on a lined paper. In fact, I have been told that I have a really irritating habit of drawing my own data sheets on lined paper or raiding the teacher’s stash of colored paper and a hunting down a ruler to guide my data collection. I abhor writing tic marks on a line without a clear label above it, and I loathe writing longhand something that can be represented by an extremely simple alpha-numeric code.

Furthermore, since the teacher and the specialist are using different forms, it is virtually impossible for them to collect the same data from the same student at the same time. The data sheets will always influence data collection. That is why I took the time a few years back to make my own data sheets that help me collect exactly the data I want to collect for a given behavior (We will get to those in a bit, they are really the point of this post).

Team members are not discussing the who, what, when, where, or why of data collection

This goes hand in hand with the above. Everyone on the team needs to be precisely on the same page regarding student behavior. I know it is vogue to use the Antecedent-Behavior-Consequence  (A-B-C) model to describe behavior (here), but I think that is not specifically useful for teachers. I like “Five Ws” questions better (cf., https://en.m.wikipedia.org/wiki/Five_Ws). In evaluating student behavior I recommend omitting the Why when still in the data collection phase since anticipating the function of behavior may lead us to miss more obvious reasons underlying behavior:

Who was involved?

What happened?

Where did it take place?

When did it take place?

Why did that happen?

 . . .

How did it happen?

Each question should have a factual answer — facts necessary to include for a report to be considered complete. Importantly, none of these questions can be answered with a simple “yes” or “no”.

Let’s think about this for a moment, compare the Antecedent-Behavior-Consequence model to the Five Ws.

On an intuitive level, the Five Ws are easier for a non-ABA trained behaviorist to understand. When we are in a hurry, we as behaviorist often forget that not everyone is trained the same way we are. And teachers usually are left confused and struggling to keep up with us as we talk about their students. So we think that because the A-B-C model has fewer questions it has to correct because it is faster.

An additional issue that follows from this approach is that behaviorists and interventionists tell the teacher and parents that (for example), “This is an attention maintained behavior, so collect data on the behavior and do not give them attention”.

We make these mistakes to the peril of the student.

Now imagine trying to keep data on a behavior we do not understand and struggle to describe. Not fun.

Now for Solutions

Always the advocate of simple solutions here is mine. I have blogged about it before (here and here). I named it a “Behavioral First Aid Kit” because it helps the classroom to heal when there are challenging or difficult behaviors in the classroom. I am an avid believer that if we collect the correct data the behavioral interventions that are necessary to help the student become apparent and it is on us as teachers and professionals to guide the student on whatever remediation is supported by the data.

I have it available at Teachers Pay Teachers, and I have provided a full watermarked preview available for you to test drive before you spend any money. I am also willing to work with anyone who needs a solution but lacks experience/resources to design what they need on their own, just send me an email from my contact page.

To address the above issues, I will describe my approach to helping teachers when I am called in to help intervene with a student. I suggest this or a similar approach be adopted.


The first thing I do is I come into the class and I use a data sheet available (here). I use this as a first pass datasheet that collects a large array of data about the entire environment surrounding the student. These data range from student time on task to teacher positive to negative comment ratios. This sheet describes the environment the student is in as well as how they interact with the environment. The datasheet also has a section for assessing if the student Has the necessary skills to behave appropriately and chooses not to or whether the student lacks the necessary skills to behave appropriately (Original survey comes from the book Lost At School by Ross Greene, which I have discussed at length here). It also involves a full teacher debrief and plan for data collection and further intervention planning.

The second day (or first day in the afternoon if I did the above step in the morning), I trade places with the teacher. I will work with the student in a small group or one-on-one and give them the datasheet I used and I let them collect data on the student as well as on me. This lets the teacher step back and observe the behavior without the stress of having to deal with it or intervene. Then I let the teacher debrief me and we work together on a plan for data collection. This always ends the same way, I break out my pack of data sheets and we select which datasheets to use for the student and I explain exactly how to use them.

The third day, I supervise and help the teacher to properly use the datasheet and to properly identify/characterize the behavior. This way we are on the same page and we know we are talking about the same thing when we discuss the behavior. At that time I will also meet with the teacher and the paraeducators and support the teacher as they explain the plan to their team.


Explicit Data Collection Training

Another critical aspect of data collection is that everyone is explicitly trained in how to use the datasheet.

A few examples of why training is important:

How would you collect data on this data sheet?

ABC_Datasheet_and_Scatterplot_Page_1
Come to think of it, how do we define the labels?

How about this one?

Classroom_Behavior_Record
Good luck figuring this one out without a cheat sheet

Without training and a number of definitions, I would not assume anyone has the pre-requisite skills to know how to use these data sheets. They might intuitively understand what I am asking for, but the datasheet itself is unclear.

Now, as part of my approach, I have two separate solutions for this problem. Firstly, I always include a narrative description of why each datasheet is to be used and why I chose it, along with what type of data should be collected on it. I also hand everyone I work with a quick data collection guide that helps to determine what type of data needs to be collected and what purpose data collection serves.

I also go into classrooms and, as mentioned earlier, specifically model and demonstrate how to use the datasheet, how we define the behavior (with inclusion and exclusion criteria), and how we can quickly mark notes for something we are not collecting data on but might want to know (i.e.,  a new behavior cropping up).

I developed this strategy after spending the better part of 15 years of my life as an academic scientist teaching high school and undergraduate students how to properly and fastidiously collect data on rodent and primate behavior. I learned during this time that it was up to me to explicitly teach (via direct instruction) how to collect data and how to see behavior the same way I do. Assuming this from even the most bright and intuitive students always led to lost time and incomplete data. However, all students, even those that struggled at first, could be trained to be experts in data collection and behavioral analysis if I allocated the necessary time to train them.

Applying the solution

For a teacher seeking to improve their data collection in class, here is my basic flowchart with images of datasheets from my Behavioral First Aid Kit.

  1. Identify the behavior of interest using a classroom observation sheet where you can identify any and all behaviors seen during the day:

    Classroom_Behavior_Record
    This looks complicated, but it is easy to use once you get used to it, especially when you need to define your own behaviors and keep track
  2. Characterize the frequency of the behavior if there is a discrete onset and offset of the behavior using a frequency data sheet.

    Interval_Recording.jpg
    This is useful for any type of time sampling you need to do
  3. Characterize the duration of the behavior if there is a discrete onset and offset (this can be done by recording the onset and offset times in the step above.

    Time_by_Frequency_Datasheet.jpg
    This is useful for ascertaining both duration and frequency of behaviors
  4. If there is an element of trying to postpone a task, quantify how long it takes for the student to start working from the time the task was presented (this is “latency to start task”, I call it procrastination).

    Procrastination_Datasheet.jpg
    This is for our chronic pencil sharpeners, arguers, and complainers
  5. If there is a clear trigger, then a simple checklist A-B-C chart or else a fill in the blank A-B-C chart can help to characterize the trigger.
    ABC_Improved_FIllable.jpg
    This type of checklist is great when it is necessary to mark data quickly

    ABC_Recording_Sheet.jpg
    This is an example of a guided A-B-C chart that follows the “Five W’s” described earlier
  6. When a trigger is identified, talk to the parents and fill out a survey on student preferences and behaviors to help guide intervention (Greg Hanley calls this an IISCA – available here – top section under Assessments and my blog on the IISC method is here)
  7. Design an intervention with the ENTIRE team, principal or school director, behavior specialists if necessary, student themselves, and parents. If everyone is on the same page as to the Five Ws of behavior and why the behavior needs to stop there is a much higher probability of intervention success

Behavioral_Plan.jpg
I often let the student provide the definitions of what their behavior is…usually by watching a video of their behavior.

Helpful tip: Always include the student in the planning step of an intervention. That way they know what you are doing and they cannot accuse you of manipulating them if they designed the plan in the first place. I have even laminated the plan and given them a copy – laminated because they will often try to tear it up in a moment of frustration

Conclusion

In my experience, the biggest problem we see in special education (and general education classrooms) is either a lack of data collection or data collection on the wrong behaviors. Often these data appear useful, but they are reporting on at best ancillary behaviors.

 

In general, if we can get on the same page with each other regarding a student’s behavior and share data sheets, both our lives and the student’s life will be made easier. We will know precisely what is going on and have the correct data to design an intervention supported by data and not just supposition.

 

Advertisements

2 thoughts on “To Help Students Succeed We Need to Collect the RIGHT Data

    1. Dr. H was my daughters aide. Exceptional! I wish he could have stayed with her. Any person who gets him is lucky, he has the ability to make the child feel good. I can’t thank him enough.

      Like

I would love to hear your thoughts on this!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s