Hans-Christian Matthiesen: When I was judging the CDI in Le Crozet in August, I had a meeting with Dr Hugues Mercier together with David Stickland and Raphael Saleh. The idea and the agenda for meeting came after some emailing and questions from both sides – IDOC and Dr Mercier. Dr Mercier had worked on some of the background material for the FEI Working Group Report – especially the judging part. Dr Mercier and the University questioned the monitoring of judges and the judging in general. I asked Mr Stickland and Mr Saleh to join me, in order to explain: what we are already doing in dressage ! Mr Stickland explained how we are monitoring the quality of judging. I tried to explain some of the ”dynamics” in judging and why we, as judges group, tend not to agree on the lower and higher marks. We explained about the CoP and many other aspects of judging. I had Dr Mercier ”sitting-in” with me in the Grand Prix Special, again to show him ”judging in real life”.
Following I had this email from Dr. Mercier with interesting conclusions. Mostly because we are already working on most of his recommendations.
All in all: positive – and I think that not only the FEI should recognize this, but the entire dressage world.
Email from: Dr Hugues MercierMaître-assistant,
Institutes of Computer Science and MathematicsUniversité de Neuchâtel
“It was great to meet you at Le Crozet. Thanks again for taking the time to meet with me, and to show me more details about how you evaluate performances in practice.
My conclusion is that most of what you do in dressage is excellent. Judges do not talk to each other, the order of the riders is partly randomised, you give marks for each movement instead of a global appreciation, you use firewalls, …
My recommendations are quite simple to state, but harder to implement
1 – Monitor the judges longitudinally. From what David Stickland showed me, the software he developed works well. Use it.
2 – Propose education / training measures for mediocre judges, so that they can improve, and get rid of the worst judges when they do not improve. From what David told me, the worst judges every year are always the same. This does not surprise me. These judges should not be allowed to judge, period.
3 – Judges do not agree on what is a good performance. No other sport has this feature, and despite what I heard from your colleagues, it is not because dressage is difficult to judge compared to other sports, or that judges rarely see the best performances and have no basis to evaluate them. This is the same for other sports: gymnastics judges rarely see the best athletes. There is something else more complicated at play. The FEI must rework the codes of points to make them more objective, which is currently not the case. What is the difference between a 7.5 and an 8.0 for a specific movement? Between an 8.0 and an 8.5 for another movement? This must be more clear than it already is. As I understand this is already in progress.
4 – It has been demonstrated that showing the judges scores during a competition induces conformity bias in gymnastics: open feedback causes judges to adapt their marks to those of the other judges of the panel. Boen & al. observed this in gymnastics in 2008 (see the reference in y first paper). Essentially, if a judge sees after three riders that he/she is always 5% lower, he/she will have the tendency to increase his/her scores so that he/she does not stand out. One solution is to hide the judges’ marks until the competition round is over. This is not fun, because spectators are curious to see these marks live during competitions, and it would be hard to hide them from judges if the spectators have them. Once again, the best way to counter this is to be as precise as possible on how to evaluate each movement, so that you can base your evaluation on clear anchor points, and not on what the other judges do.”