Ed Focus Initiative

Error Analysis to Get Beneath the Scores

Another EdFOCUS technique is to help school staffs look for error patterns. Once they determine which concepts have proven the most problematic and for which particular students, school staffs need to drill down to the individual test items comprising that concept and look for the types of errors made.

For multiple choice items, the distractors chosen by each student provides insight into his and her level of understanding. When a majority of students selected the correct answer, effectively-constructed test items reveal a level of understanding and mastery that can and should be replicated. This “correct” answer also informs the re-teaching or intervention that must be provided to students who did not make the correct choice. Similarly, when a critical mass of students selects the same wrong answer, teachers should determine what misunderstanding that represents and use it as the basis for remediation. In most cases, when several students select one or more wrong answers, the concept should be re-taught. But the incorrect answers - if they are properly constructed - are just as instructive to teachers as the correct answers. To really take advantage of the testing situation, smart practitioners want to know as much about what students were thinking as possible.

EdFOCUS also strongly suggests that each grade level team identify the 5-10 concepts that need to be addressed as well and 5-10 concepts that the same students understand very well. As the teams discuss these two extremes, it is important to distinguish the kinds of instruction used for the “green” concepts from that used with the “red.” This exercise is very important to give teachers confidence that they can parlay their successes to turn-around their frustrations. More importantly, they can see the connections between the teaching strategies and student responses to determine the impact of teacher-effects on their students’ test performance.

Once the types of errors are discovered and any patterns detected by grade-level or AYP group, the school staff should next turn to the curriculum and ask several driving questions:

  1. Where do the concepts appear in the Curriculum Map or Pacing Guide? Is the full text of what is expected of students apparent in the Map or Guide, or is just the concept (e.g., rational numbers or setting) listed. Are the concepts repeated across the school year, or are they taught one time only?
  2. What does the test item ask students to do? Many of the high stakes test questions are contextualized with real-world scenarios and events. If students have not been exposed to this in the teaching, they will not even realize that the concept is the same thing that was done in class. It is important to understand to what cognitive level of thinking the questions are constructed. Many workbooks and practice materials are more “drill” and “skill” oriented, but do not give students an opportunity to apply what they have learned. If the standards expect application, then rote level memory will not suffice for students to be successful.
  3. Look at the predominant mistakes made by students and determine which students made the mistakes. Did the best performing students miss the same questions as the poor performing students? Who is making the errors and the type of errors made informs the team as to what interventions are needed going forward. In addition, how did Students With Disabilities perform on these items.
  4. In terms of the priority needs, teaches need to compare how the concept is taught in comparison to what is being asked of students on high stakes tests. Are students being given the opportunity to master the concept at the level and depth required in the standards? Does the test match the standard in terms of this cognitive demand?
  5. What will be different going forward in the instructional delivery and practice? What will be done for students in the various RTI levels? Are there different materials and resources that must be obtained to offer the instruction needed? What Professional Development is needed for concepts where teachers feel they need some additional support?