First round of FSA test scores released …. sort of
6/26/2015
UPDATE 10/1/15: Since the publication of this post, additional FLDOE document releases and the Alpine Testing Solutions audit report have clarified that the release timing and linking methodology decisions associated with these preliminary Algebra I and 10th grade ELA scores were driven by DOE efforts to comply with statutory reporting requirements. For more updated information on remaining steps in the state's standard setting process and reporting plan, see http://www.jaxpef.org/news/what-does-validity-mean-and-other-questions-you-have-about-the-current-state-of-school-accountability/.
The Florida Department of Education today released 2015 Algebra I and 10th grade English/Language Arts FSA test score results, the first piece of information to be released on how Florida students did this year on the much-debated new assessments which replaced the previous FCAT 2.0 assessments.
The key takeaway for how students both locally and statewide did is … we’re still not really sure. That’s because the final “cut scores”, or passing requirements, of the new tests have not been established yet and are not expected to be for several months. The results released today are based on a statistical approximation technique called “equipercentile linking”, which uses statewide distributions of student scores from last year’s FCAT 2.0 test results and applies them to this year’s corresponding FSA assessments. Student scores this year are then referenced and scored relative to each other and those distribution points from there.
To be more specific, 55% of students statewide passed the 2014 10th grade FCAT 2.0 Reading assessment last year. Because passing-level cut scores have not been yet been established for the 2015 10th grade FSA English/Language Arts assessment, the results released today were approximated by finding the score at which 55% of students on this year’s test were above and making that the “passing score”, and evaluating everyone else relative to that.
One way to think about it is that it’s sort of (but not exactly) like “grading on a curve”, as opposed to grading students against a set standard – which the test is intended to do.
This approach raises a number of potential concerns to be aware of in interpreting any scores released today or any time before FSA-specific grade-level cut scores are established. Foremost is that the FCAT 2.0 and FSA are different tests, potentially very different. Where previous year’s FCAT 2.0 Reading assessments focused on just reading, the FSAs have shifted to English/Language Arts assessments – including a writing component incorporated with the reading items (as opposed to tested separately as they were in previous years). Why that makes scores based on equipercentile linking potentially problematic is that they are based on the relating students’ abilities to read and write this year with comparable students’ abilities to just read last year.
In addition to creating potentially problematic cut scores when used between different tests, another concern is that the equipercentile linking method can also put some degree of artificial constraint on year-to-year changes in school or district scores. Changes will still be apparent at the district and school levels because cut scores are established at the statewide level, and districts or schools may still do better or worse relative to each other than they did last year. But ultimately there is a restriction on the number of districts or schools who can be “failing”, which may mask at this point how much better or worse they actually fared on the standards of this specific test.
Another way to think about it is that, using equipercentile linking, the state could have released all statewide FSA grade-level passing rates for this year before they gave the assessments. The statewide passing rates are based on last year’s, they are unchanged by design – though individual districts, schools or students may shift somewhat within them.
A third concern is that releasing the scores now based on equipercentile linking is likely to cause confusion if scores are re-released – and significantly different – in a few months when FSA-specific cut scores are established. It is not clear yet how the FDOE intends to handle the rescoring of assessments at that point, though a similar approach during the transition from FCAT to FCAT 2.0 test scores a few years ago resulted in significant confusion.
It should be noted that waiting until after a new test is administered for the first time to establish appropriate cut scores or other properties (particularly without sufficient field-testing) is not inappropriate or uncommon. To some degree, students’ performance relative to each other in the first large-scale administration of the test will influence where appropriate levels are set – in conjunction with input from content-area experts, psychometricians, and others. This is a common and accepted component of test specification and validation.
Why today’s preliminary set of scores would be released prior to the completion of that process is the more important question. In all likelihood the release today is a good-faith effort on the part of the FDOE, and requests from districts, to provide districts and schools with planning information to be making determinations about school and student needs moving into next year. Knowing the FSA validation and cut score establishment process will not be completed until several months into the 2015-2016 school year, the presumptive motivation for this release would be that some planning information estimates are better than none.
Ultimately, this may or may not be true. It could very well be the case that the FCAT 2.0 and FSA are similar enough that scores released today are very similar to those eventually established as a result of the cut score development process – and therefore helpful as a general planning guide for school and district leaders. The concern is that we do not know if that’s the case yet, and so would caution school and district leaders, parents, community members and media to interpret the scores released today very carefully and with the understanding that they may change significantly in a few months.
For what it’s worth, 48% of 10th graders in Duval County Public Schools passed the FSA English/Language Arts test according to the scores released today (same as last year), and 57% of DCPS students passed the Algebra 1 EOC (down slightly from 58% last year). Statewide, scores in this release remained the same as last year (~55% passing 10th grade ELA, 67% passing Algebra 1 EOC) – as would be expected with equipercentile linking.
Full district-level results for each test can be found here and here.