Advances in teacher value-added modeling measures
10/18/2013
In 2011, the Florida legislature passed a law requiring that teachers in Florida be evaluated in part using student achievement data. Teachers are measured based on a statistical inference of how much they impacted a student's learning over the course of the year. This is done using a method called "value-added modeling".
In value-added modeling a formula is used to predict how well a student would be expected to perform on an assessment, based on previous assessment performances and student, class & school characteristics. That prediction is then compared with how well the student actually performs. The difference between how the student would have been predicted to do based on past performance and specific characteristics alone and how he or she actually performed is considered the "value-added" by the teacher to that student's learning. So if a child exceeds expectations, that is considered evidence of positive value-added to the student's learning by the teacher that year.
The Florida formula for calculating predicted student achievement is a complicated one, but takes into account the following factors:
- The number of subject-relevant courses in which the student is enrolled
- Two prior years of achievement scores
- Students with Disabilities (SWD) status
- English language learner (ELL) status
- Gifted status
- Attendance
- Mobility (number of transitions)
- Difference from modal age in grade (as an indicator of retention)
- Class size
- Homogeneity of entering test scores in the class
In essence, the model estimates the impact of these factors by looking at how growth was affected across all students with any of these characteristics, and then adjusting the expected performance of any student with that characteristic accordingly. (A full technical explanation of the model is available here.)
Using enough student data, these models can actually predict well how a student with specific performance history and characteristics would be expected to perform.
The key issues are making sure the model correctly accounts for everything it needs to, and that the data is all accurate. One of the common concerns about value-added modeling is that there are multiple different ways the models can be constructed. For example, should you control for student poverty or not? Is it better to use three years of prior data than two, or does it not make a difference? With each potentially important element that is added or not added, a change is being made that could ultimately affect some individual teachers differently.
Last week, the Institute of Education Sciences and Michigan State University's Education Policy Center held a conference bringing together some of the top value-added modeling researchers, education policy leaders and educators from around the country to share ideas on the latest advances in not only statistical designs but also practical implementation strategies.
We attended to learn more about the advances being made that Duval County and Florida can learn from to continue trying to make sure we have the most comprehensive, accurate and fair teacher accountability measures possible.
Some of the key issues raised at the conference that Florida may want to consider as part of ongoing efforts to ensure the accuracy of our teacher evaluation models:
- Improved roster verification processes: One of the biggest challenges for many states is making sure the right students are matched to the right teachers in the data. Because students move between teachers, schools and districts regularly, sometimes data records are not completely up-to-date in student-teacher assignments when VAM scores are calculated. Processes for having teachers personally check, and principals cross-check, which students they taught in which subject areas were shown to significantly reduce the risk of teachers being incorrectly classified (as "Highly effective", "Effective" or "Needs Improvement") in their final results.
- Improved 'dosage' verification processes: Related to correct student/teacher/course associations was the issue of how much a teacher had a student for any individual lesson. For example, in some states elementary school students are assigned to their homeroom teacher in the data for all their primary subject areas (i.e., Reading, Math, etc.). But many of those schools actually use "team teaching" or other instructional models that have students going to different teachers for all or some of their instruction. As part of the roster verification process mentioned above, some places ask teachers to assign a percentage of the students instruction in each course area that they are responsible for. For example, 100% if they are the students only Reading teacher, 50% if the student spends equal time with two different teachers for Reading instruction, etc.. Again, in these models principals are often asked to verify and correct when student's assignment numbers between teachers do not add up to 100%.
- Multiple model formulas: Many states use a single, uniform statistical formula for measuring teacher-value-added effect across all grade levels. But the way a student's instruction is structured in elementary school is typically quite different from the way it is in high school. For example, some researchers at the conference presented evidence on the impact of sorting tracks in the upper grades (college-prep, career and technical ed, other) on teacher value-added estimates when not accounted for in the model. This raises the possibility that, rather than a single formula for teachers at all grade levels, perhaps a formula with multiple variations specifically-tailored to the grade-levels being measured would be more appropriate.
- Improving school effect attribution: One of the core functions of any value-added model is to separate what impacts the school environment as a whole has on achievement from what impacts the teacher specifically has within the classroom. (For more explanation, see here.) But what to do with that is another question. In Florida, 50% of the school effect is credited to the teacher, on the thinking that all teachers share at least partial responsibility for the overall school environment. But by assigning 50% of the school effect to each teacher, that is telling the model that each individual teacher is single-handedly responsible for half of the impact of the entire school. In other words, this might make sense if there were only two teachers at a school, but in general it's assigning too much credit and blame to each teacher for whatever else is happening in the school as a whole. Many researchers and policy makers talked about options they are using that makes more efficient and appropriate use of the teacher and school components together.
- Better connecting research to policy: Some of the advanced VAM researchers at the conference lamented the difficulty of getting their more complex, advanced (and often more appropriate) models for measuring teacher value-added effect adopted by politicians or education policy makers who may feel constrained by the need to be able to explain an idea simply to their constituents. In other cases, limitations placed on models prior to developing them were also cause for concern among some researchers.
Measuring teacher effectiveness is a critical component for ensuring all students have access to highly effective teachers, capable of accelerating their learning at every grade level. But it is also a highly complicated undertaking. Value-added modeling is a highly useful method for measuring teacher's impacts on student learning when statistical models are specified well and results are interpreted strictly. There are some tremendous advances in value-added modeling happening around the country that Florida should try to learn from and evaluate what elements are most appropriate for here.
For more information about Florida's value-added model formula click here, or to view recordings of all the presentations from the conference, click here.