PPS and ODE Fudging the Numbers

PPS recent interest in equity and closing the achievement gap has taken me on a walk down memory lane. 

I may not have given PPS credit where credit is due.  They have invested a lot of time and resources into hiding pitiful student performance and the achievement gap over the last several decades.  

My dates may be a little off because I’m not certain when some of their tricks began but I have documentation confirming the dates listed below:

1986-1998 Clear and Intact – PPS only included the scores of students who were “Clear and Intact” when reporting student achievement results.  Clear and Intact students were students who were tested and in the same school for two consecutive years.  The achievement levels of poor students who were more mobile were often left out of the results.

1998 PPS/ODE Change  – PPS and ODE changed cutoff scores to align the Portland Achievement Levels Tests to Oregon Department of Education (ODE) tests.

June 1999 Ed-Flex Waiver –  Oregon Department of Education (ODE) approved PPS request for an Ed-Flex waiver which allowed PPS to use their own definition for Adequate Yearly Progress.  The district claimed that their definition was more rigorous than ODEs but it was actually the lowest standard in the state.  PPS definition only required schools to have 60% of the students at just one grade level to have met benchmark in math or reading in a three year period.  Portland Public Schools increased the percentage of schools reported to be making Adequate Yearly Progress from 55% to 90% by making the change.

1999 to current  Levels Tests – While most people believe that there are standardized tests in Oregon, there are actually multiple levels of the state tests.  The levels tests were developed in Portland and adopted by the Oregon Department of Education.  PPS has been using multiple levels tests since at least 1999.  ODE Level Tests: Questions and Answers (December 1999) states “There are three levels of the test-Levels A, B, and C.  Level A is the lowest level of difficulty.  Level C is the highest.”

Minority students have been disproportionately given easier tests than white students.  District staff have argued that students would achieve the same score regardless of levels but according to PPS Spring 2001 Individual Student Report, “when a test is too easy for a student, students get high RIT scores.”

Spring 2001 Individual Student Report – The Individual Student Report included an “Important Note – If your child had a third grade reading RIT score of 225 or above in 2000, he or she may have a fourth grade score that is the same as, or even lower than, the third grade score.  The Oregon Department of Education recently announced that the third grade reading test used in 2000 may have been too easy for the highest achieving children in the state.  As a result, scores for this group of students may have been artificially inflated last year.” 

2001 School Report Cards – ODE counted “conditionally meets” scores as “meets” when reporting on the percentage of students meeting state benchmarks on the district and school report cards.

 September 2007 PPS/ODE Change - A panel of teachers, professors, principals, business leaders and others reviewed Oregon’s reading, writing and math tests to determine whether they were hard enough but not too hard for students in each grade.  They decided that the math and reading tests were too easy in grades kindergarten through seven, math test too difficult in grade eight and both reading and math tests too difficult at the high school level.  They raised or lowered the scores needed to meet benchmarks to reflect the review.

2007/08 to current OAKS Computerized Levels Tests - Oregon tests students in reading, math, and science using OAKS.  The OAKS online assessment is an adaptive assessment, which means that the items presented to the student vary in difficulty based on the student’s performance on the previous item.  Therefore, the state creates a grade-level item pool rather than a single pre-made test for each grade level. The computer selects questions based on the answer a student gives to a test item, which in turn determines the difficulty of the next item that the computer will select.  All students may take the tests three times per year retaining their highest score.  The OAKS assessment is just a computerized version of the levels test.

2008/09 School Report Cards – Jefferson High School’s Report Card included the test results for the Jefferson Young Women’s Academy.  Jefferson is the only high school to include middle school results on the report card.  They don’t tell you they’ve done that.  You have to compare it to ODE’s 10th grade assessment results.  The inclusion of the middle school scores significantly increased the percentage of Jefferson students meeting benchmarks in reading and math. 

The Jefferson Report Card listed 58.3% of the students meeting reading benchmarks but just 39.8% were 10th graders.  In math, Jefferson reported 41.5% meeting but just 17.9% were 10th grade students.

2007/08 Margin of Error - PPS has an unusual definition of “margin of error” for Adequate Yearly Progress purposes.  Normally people use margin of error to measure the precision of sample data.  In PPS case, there aren’t any samples because the numbers reported are the actual population totals.  Instead of adding or subtracting the margin of error, PPS only ADDS it to the percentage of students meeting benchmarks.  This is done districtwide.  At Jefferson, 8.5% of the limited English students met English/Language Arts benchmarks but PPS added a margin of error of 23.55% making it appear as if 32.06% of the students met benchmarks.

On March 8, 2010, State Schools Superintendent Susan Castillo named fifteen Oregon public schools that made significant progress in closing the achievement gap. 

Selected schools made it through a data screen that identified schools where student subpopulations (minority groups, students with limited English, special education students, etc.) made significant progress compared to the comparison groups (white students, English speaking students, non-special education, etc.). Student cohorts were analyzed using data from 2004-05 through 2008-09. A team reviewed the data and examined both school to state comparisons and within school comparisons. Statewide report card and AYP data was also analyzed. Schools demonstrating the strongest subpopulation growth (and hence the most progress in closing the achievement gap) were invited to submit an application for further review. A Blue Ribbon Panel composed of educators, business leaders, Youth Advisory Team student members and community members reviewed the data and the applications and made recommendations to Superintendent Castillo.

How could the Blue Ribbon Panel have reviewed 2004/05 to 2008/09 data and identified improvement when parents are cautioned against comparing results from those years?  The panel members would have to have been math geniuses to wade through the mess that PPS created but my guess is that the Blue Ribbon Panel wasn’t provided with this background information.

AYP – Another Year Passes in Oregon.