IPI Classroom Observation Process Overview

When the data collector enters the classroom to make a student engagement code, the observer must immediately make a mental snapshot of how students appear to be engaged in learning.  The observer then moves among the students, looking at their work, interacting with the students and the teacher as necessary, to determine which IPI code accurately describes how the majority of the students were engaged in learning upon the observer's entry into the learning setting.  The observer is typically in the classroom two to three minutes, during which the observer notes the total number of students and the number who were engaged in various activities and the appropriate code for those students.  When the data collector steps outside the learning setting, the appropriate IPI Category code is recorded on the data recording form or app.  The data collector then moves to the next classroom/leaning setting and repeats the process. 

The IPI Process is based upon accurate data collections.   That means the data collector must be skilled and endorsed to collect IPI engagement data.   Therefore, IPI data collectors must complete an IPI Workshop and the IPI Coding Assessment with a coding score that meets the necessary standard for endorsement as a data collector.  Coding with accuracy and consistent accuracy requires significant learning experiences so the data collector is skilled enough to assess a multitude of learning scenarios and determine how students are cognitively engaged in each type of scenario they might encounter.  The following sections provide insight about coder validity and reliability as well as the practical steps a school will commonly use to best collect IPI engagement data.  All of the details about the data collection process cannot be covered on a website, but the following will address most of the typical questions and provide a basic understanding of the process and how the data collectors are prepared via the IPI Workshop.   

 

Data Collector Accuracy

Collecting accurate data about student engagement is obviously the foundation for understanding how students are engaged.  The data profiles from the collection of quality data provide the basis for the collaborative faculty study of the data.  Since the inception of the IPI, the data collection process has been guided by protocols to increase the likelihood that the data collected will be accurate.  And because the process is grounded in the belief that teachers conduct multiple observations and that multiple observers contribute to the data pool, the issues of consistent accuracy and cross-observer accuracy are also important.   Therefore, since the earliest of IPI Workshops an emphasis has been placed on data collection validity, reliability, and inter-rater reliability.  In the IPI Process, those three terms are defined as follows:

Data Collection Validity is the data collector's accuracy against the six IPI categories.  High validity means the data collector is accurately coding the classroom engagement observations per the six categories.  So validity, as used in the IPI Process and explained in an IPI Workshop, means accuracy of data collection.  At the conclusion of each IPI Workshop, the participants take an IPI Coding Assessment.  This assessment is used to provide feedback about the participant's coding accuracy.  An Assessment Score of .80 or higher is necessary for permission to be a data collector in a school setting, thus collecting the data for the purpose of school/instructional improvement.  The participants feedback letter indicates whether they have met the standard and if not, provides suggestions about how to increase their coding accuracy and options for retakign the assessment.  Also, a standard of .90 is necessary for permission to collect IPI data for research purposes.  Over the years, the standard has been even higher (.95) for university staff and graduate  assistants who collected data for our university-based research.    

Data Collection Reliability is the data collector's accuracy across multiple similar observations.  So, reliability, as used in the IPI Process and explained in the IPI workshop means data collection accuracy over time for observations that should have the same code.  In other words, when a data collector sees student engagement of a particular type at 8:15 AM and then sees the same type of engagement at 10:20 AM, the observer is making the same (correct) code for the two scenarios.  During an IPI Workshop, participants complete between 40 and 50 codes and do so with scenarios that range from vastly different to highly similar, across different classroom learning contexts and/or grades levels.  That number and variety of the scenarios is necessary to establish the coder's consistent competence. 

Data Collection Inter-rater Reliability is the data collector's accuracy against other data collectors.  The process for developing a data collector's validity, reliability, and inter-rater reliability is evident throughout an IPI Workshop.  For example, participants' codes for the learning scenarios and for the practice observations in real classrooms are determined independently and then those codes are shared by each participant with all other workshop participants.  The scenario codes are then discussed with colleagues, enhancing the learning process and making transparent the development of inter-rater reliability to all participants.  As has been noted by countless workshop participants: "the degree to which we differed when we started the day (workshop) and the degree to which we agreed by the end of the (workshop) day was miraculous."  The IPI Workshop is specifically designed so participants not only realize that they are all growing in their coding skill capacity, but it is also designed to help them realize that they are growing together and building inter-rater reliability as they work through the day-long workshop.  That is an important realization for all data collectors because they must have confidence that their colleagues who are collecting data are coding just as accurately as they are throughout the school day. 

 

Collecting a Systematic, Proportionate Sample of Engaged Learning

The procedures used by the IPI Team to collect data are explained in the IPI Workshop as a systematic, proportionate sample of student engagement.  That means that the school data collector will begin at a random classroom and then move systematically throughout the school following a logical and purposeful pattern that will lead to observations in all classrooms before the pattern is repeated.  The pattern is repeated again and again throughout the entire school day.   As noted, the only time the data collector is not busy making a code is during the first five and last five minutes of the learning period (designated time for learning in the elementary setting and class period in the middle or high school setting).  In so doing, the classroom observations of student engagement are proportionate, with all classes in the data pool equivalent to their occurence during the school day.  If the school has alternating day classes, with approximately half of the curriculum on one day and the other half on the next, then the data collection process should be over two days, so the profile represents data across the entire curriculum. 

The IPI Team members who meet the data collection standard of .80 or higher accuracy on the IPI Workshop Coding Assessment serve as the data collectors for the school.  In a school with an enrollment of 700-800 or fewer students, three members of the IPI Team should take turns collecting data, usually in shifts of about 2 hours each.  Using that pattern, the three data collectors can collect a pool of about 120 to 140 data points in an elementary school and about 140-160 data points in middle school and high school.  That would represent at least three or four data points per classroom by the end of the day.  In a larger school, the number of data collectors would double, with two data collectors working simultaneously, but not together, for the first third of the day, then another pair would work the next third, and another pair would finish the day.  That works unless the school had more than 1500-1600 students.  As the school increases in size, the number of data collectors must also increase to maintain the ratio of at least three or four data points per the number of classes in session during a designated class time/period.  Using the pattern of three shifts for the data collectors allows the data collectors to be back with their students for 2/3 of the school day...thus their students are in the data pool with them as the teacher, rather than with a substitute teacher for at least 2/3 of the day.  

 

Determining the Number of Data Collection Days Per School Year

Typically, a school's IPI Team selects three, four, or five days across the school year.  Based on what we learned in a major longitudinal study of the data and countless anecdotal comments from schools, four data collections a year is the goal toward which a school should strive.  However, it should be noted that the number of observations alone is of no value...each data collection day must be followed in a timely manner (within a week or two) with the faculty collaborative learning conversations--study of the data. 

When we first began using the IPI Process for school improvement purposes, it seemed logical that a school would need to collect and study their data at least three or four times a year.  However, that was only a supposition based on logical practices of school improvement and change.  After studying several years of data, it became apparent that the early supposition was correct.  What we learned from our studies over the years was:

A single data collection and faculty collaborative study of the data each year was equivalent to a "waste of everyone's time."  There seems to be no noticeable change in instructional design and student engagement across a school and no statistical changes in student academic success from one data collection/study each year.

Two data collections followed by faculty collaborative study each year were associated with a minimal level of faculty awareness of about student engagement, but little change in practice across the school.  Also, there was no clear evidence that two collections and discussions were linked to enhanced student achievement.

Three data collections with faculty collaborative study in a timely manner produced positive changes in instructional practices that fostered more student engagement and higher levels of thinking during engagement.  Three data collections a year is the minimal number an IPI Team should work toward.

Four data collections a school year with faculty collaborative study of the data profiles seems to be optimum, based on what we have learned in the past decade of observing and studying the process in hundreds of schools across the country.  And for what it is worth, five might be slightly better but tangible evidence to support that has not been found.  Some schools even collect IPI data monthly, but they do not seem to be any further along with meaningful changes than are the schools that collect data four times a year.  The key is in the faculty collaborative study and what we have noticed is that collecting data monthly does not seem to be linked to an equivalent number of meaningful faculty collaborative discussions of the data.

 

Selecting the Observation Dates

Some schools' IPI Teams work with the principal to identify observation dates and place their IPI observation dates on the school calendar at the beginning of the school year.  They also place on the calendar the date of the whole-faculty study of their data profiles.  This practice seems to work for most schools because the faculty know the observation dates and the schools generally follow through and make their data collections as scheduled.  While that is positive, the more positive factor is that those schools tend to also study their data within a few days after the data collection.  Remember: studying the data in a timely manner is an important ingredient to change.  Other schools' IPI Teams simply decide they will collect data each quarter and select those dates a week or two ahead of the observation date...and they try to schedule their faculty discussion in a timely manner as well.  The key, once again, is to get three or four periodic collections and then have the whole-faculty collaborative study in a timely manner. 

When determining the day for the IPI data collection, select days that are "relatively typical."  We all know each school day is unique and many things happen that cannot be predicted that disrupt the flow of a "normal" school day.  However, the IPI Team can check the school calendar when selecting the observation dates and talk with the principal and other leaders to ensure the day has a chance to be a rather typical school day.  In other words, don't select days with abbreviated class schedules or with concerts, sporting events, or field trips that will take significant numbers of students away from the school for a portion of the day.  And in high schools and junior high/middle schools, watch for after school athletic events, dances or other social events, or other events that might detract from the natural flow of class and learning time during the school day.  When the faculty study the data, they need to do so with the understanding that the data collection day was a relatively typical school day, thus eliminating any "my kids could not focus because..." comments that might compromise the data, e.g. "my class had a hard time focusing this afternoon because of the important basketball game tonight."  The faculty need to view the data as fair and accurate, in order to be comfortable with the data and learn from the study of the data.  Also, as the school is developing an understanding of the IPI process, most high schools should avoid collecting data on a Friday.  For some reason, Fridays are too often characterized as non-typical learning days, so start the process with collection on other days and when the faculty is ready, they will request Friday data to see if engagement on Fridays is that much different from other days of the week.

 

Make the Process Transparent...Pre-Inform Faculty Re Data Collections

One or two days prior to the IPI data collection date, the IPI Team should indicate to the faculty that IPI observations are scheduled for a specific date.  This process of openness is recommended because if the IPI is to be of greatest value to the faculty, and thus to instruction, and thus to student learning, the faculty must be comfortable with the process.  Comfort with the IPI process means faculty find no reason to "jazz-up" their lessons on observation dates because they understand the process is not an evaluation of them, but a whole-school profile of engagement for all students and all classes.  And, if the faculty are comfortable with the process, they will be more open to studying and learning from the IPI data profiles.  

The IPI should be a faculty driven, faculty empowering process.  Teacher leaders should be the data collectors, they should study and organize the data before the faculty study of the data, and they should lead the faculty in a collaborative learning experience about the data and about strategies to address issues from the data.  Seldom does a faculty really learn from the data and grow in their instructional design and delivery that fosters quality engagement if they are viewing the IPI process as "administrator driven" or as "supervision or evaluation of my competence."  The IPI should be viewed by the faculty as "hold harmless" and "non-judgmental."  When faculty come to realize and eventually accept this basic premise, then the greatest amount of faculty development from the data will accrue.  The IPI Team and the Principal should do everything they can to make the process a teacher-led, transparent process that faculty will see as non-judgmental. 

If the faculty provide the typical "dog and pony show" common to announced evaluative/supervisory observations for the three or four days a year the IPI data are collected, then the faculty is studying data profiles that represent engagement on those days, but those days do not represent how students are engaged the other 176 days of the school year.  It is imperative that the IPI team make every effort to help the faculty become comfortable with the process so the study of the data can be most productive. 

 

Build Faculty Awareness and Understanding before Beginning the Process

As a result of participation in the IPI Workshop, the IPI Team and the Principal will develop an understanding of how the IPI Process is implemented.  Once they have completed the Workshop, they should meet and discuss the process and determine if they want to move forward and implement the process at their school.  Logically, the next step would be to share the process with the faculty and seek their thoughts about the IPI and its value to their school.    They should also share with the faculty and provide the faculty with the time to read and discuss the Instructional Practices Inventory Categories and the Protocols for Data Collection handouts used during the IPI Workshop.  Obviously, the IPI Workshop goes into details about the steps to collect data and to study the data as a faculty.  The IPI Team can determine how much detail they wish to share with the faculty about the process.  But at the very least, they should clarify the following items during their discussion of the IPI Process with the faculty.  Otherwise, the faculty will be ill-informed about the IPI Process and they will likely view the process as judgmental and punitive, as has too often been the case with prior efforts to enhance learning experiences for students.  The items that should be discussed are: 

The IPI Process provides an optimum picture of student engagement across all all classrooms and for all students in the school.

The IPI Process is hold-harmless and non-evaluative.

All data codes are anonymous and all data are aggregated as school-wide data.

Principals will not be data collectors nor will they see any individual data codes.

The IPI categories depict how students are thinking and engaging, not how well the teacher is teaching. 

The focus of the observation is on the students, not the teacher.

The IPI Process creates valuable data for the faculty to study and from which they can learn and consider how to better design learning for their students.

The data are collected by certified, trained data collectors who collect a systematic, proportionate sample of engagement throughout the school day.

When, occasionally, a data collector is not certain of a code, or when there is an even number of students engaged in more than one code, the data collectors need a guiding principle for making the code.  In the IPI coding process, they will assign to the higher code simply because they are creating an "optimum" picture of student engagement...the best picture of how our engagement looks that day. 

Data are not collected during the first five minutes or last five minutes of an assigned learning time or class period...students are given time to get into class and get settled into learning and even at the end of a class time, sometimes students might complete the tasks early and there is not adequate time to begin new learning, thus a five-minute grace period is provided at the end of the class period or learning time.  This accommodates differentiation of instruction and fits with the "optimum" picture of engagement.  It is also necessary because it removes the argument that "it took a few minutes to get started" and "there was too little time to begin a new learning concept." 

 

The IPI Process Treats Teachers as Professionals

All of the above concepts are implemented because it is critical that a faculty understand that the IPI is a non-punitive process so they can be comfortable openly studying and learning from the school-wide profiles.  Education has, for too long a time, demanded certain teaching practices of teachers and created monitoring and punitive practices to ensure that the expected practices were followed.  Within that perspective is the basic assumption that teachers are not professionals and must be told how to best teach their students.  For most teachers, that is not the case.  They are well educated, conscientious, and hard working professionals who care immensely about their students and their students' success.  Inherent to the IPI Process is the premise that teachers will perform as professionals who want to learn about how to best work with their students if they are treated as professionals.  The IPI empowers teacher-leaders and the whole faculty by providing them with their own data in a format that allows them to study and learn from the data.