Meeting in Trento, Italy on October 3, 2008 the ISU Council voted to reduce the number of judges used at ISU championships and at the Grand Prix final. The number of judges on a panel will now be a minimum of eight and a maximum of nine, compared to the former minimum of ten and maximum of twelve. Under this rule, nine judges will be selected for the championships this season, with two randomly selected discard judges. That leaves seven scoring judges for calculating the single trimmed means, with five marks being included in the average of each GoE and Program Component.
Coming just three months after the 2008 Congress, where it was neither discussed nor revealed to ISU members, the Council invoked Article 17 of the ISU Constitution which gives the Council the power to do anything to the scoring system it wants to without seeking the approval of the ISU members. According to our sources, many in the skating community are shocked and offended that a matter of such importance was not presented to the Congress.
Even before reducing the number of judges, the mathematics of IJS was a deal with the devil, since even in the best of economic circumstances the ISU could never afford the number of judges needed to produce reliable results of the highest integrity expected for a World or Olympic Championship. The former panel size was an unholy compromise between the need for results of the highest reliability and integrity and the cost of having enough judges to accomplish that -- with the decision made that reliability and integrity would be sacrificed to economics.
Now, to save money in the face of declining revenue, the ISU is reducing the number of officials by three at each of four championships this season. That corresponds to 12 man-weeks of travel expenses in the budget for each competition. At a generous estimated cost of $3,000 per man week, it would appear the ISU will save about $36,000 per competition, or less.
Even in these difficult times, however, $36,000 is a drop in the bucket. The ISU could probably save that amount of money by holding a few less parties at the championships, or using less expensive hotels, or leaving home unnecessary VIPs who do not contribute to the actual holding of the competitions, or flying coach, or saving paper by not publishing an endless stream of communications, or any number of other places that do not affect the integrity of the results. But no. Instead the ISU has chosen to deal with the cost of competitions by adding melamine to watered down milk. And the skaters will pay the price.
When it comes to contamination due to random and systematic errors, the IJS calculation method does not compare favorably with the 6.0 method. As a rule of thumb, the reliability of an IJS calculation with "N" judges is about as good as a 6.0 calculation with "N-2" judges (and in some circumstances less). Thus, in terms of reliability and integrity, a seven judge IJS calculation is about as good as a 4-5 judge 6.0 panel. To keep the reliability of the results that 6.0 panels provided with nine judges, IJS should be using at least eleven scoring judges. Instead the championships this years will be calculated with seven.
How, then, might this affect championship results?
For a single bad mark due to random error, misconduct, ignorance or national bias (which remains rampant under IJS), the importance of that mark is increased by roughly 40% since a bad mark will now be one of five instead of one of seven.
As an example, if a judge skews each Program Component in the Men's Free Skating by 0.25 points for one skater, that will move the point total by 0.5 points. Marking a rival skater down 0.25 points drives down that skater's score by 0.5 points. One judge, can therefore introduce a 1 point change in point difference between two rival skaters by manipulating the five program components alone.
Now add to that the opportunity to skew 13 elements. In the worst case, one judge can skew the point difference by more than two points in GoE. In principle, one judge can skew the total point difference between two skates by over 3 points.
And how often are results determined by less than 3 points? A lot!
Then there is block judging.
Under both 6.0 and IJS we have seen that a nine judge panel is frequently divided into two geo-political blocks of four and five judges each. With two of these nine judges eliminated by random draw, it would not be uncommon for the two discard judges to be from the same block. That would lead, for a significant number of panels, to seven scoring judges where two judges are from one block and five are from another. If a high mark is taken from one block and the low mark from the other, the five marks averaged will consist of one mark from one bock and four marks from the other. In this case, which will not be uncommon, the skater from the "wrong" block will be, in a word, screwed.
Or worse. Consider that with smaller panels, fewer countries will have judges on the panels, leading to panels where one block is not just a slim majority of the panel, but the vast majority of the panel.
No doubt about it, economic conditions are tough right now for skating, as they are for just about everyone else. Businesses everywhere are looking for ways to cut cost. Nevertheless, one would have thought that even the most casual of business person within the ISU would recognize you do not survive a difficult economic climate by destroying the integrity of your product.
A far better choice would have been to eliminate the use of discard judges and do the single trimmed mean on all nine judges. Since the discard judges don't do anything to help or hurt the integrity of the results, the savings from leaving three bodies at home would be realized and the integrity of the results would be no worse off than last season.
Copyright 2008 by George S. Rossano