This year, I successfully completed a four-year journey towards a Master's degree from UBC's School of Kinesiology. While I work to convert my thesis into a publishable format, I have recorded a presentation for anyone in the officiating community who is interested in this research. The presentation can be accessed here:
Summary
The most prominent conversation in the sport officiating community is whether it is possible to quantify the performance of officials. Ultimately, the best officials make the correct decision most often, but there is no accepted metric or methodology for making this determination. The purpose of this analysis was to examine 28 top ice hockey referees, officiating the International Ice Hockey Federation’s (IIHF) Men’s and Women’s World Championships, and assess the accuracy of their penalty decision-making. Their penalty calls (523 from 86 games) were assessed via video by a panel of experts and the data collected was compared with the final three-group ranking of referees (1 – Top, 2 – Middle, 3 – Bottom) at the conclusion of the championship.
There is a statistically-significant difference in the percentage of on-standard penalties called between the three groups (p = 0.001, ω2 = 0.386). The difference is not between Groups 1 and 2 (p = 0.335), but between a combined Groups 1 and 2, as compared to Group 3 (p < 0.001, d = 1.691). This analysis provides clear benchmarks: elite referees are correct in their penalty calls 82.5% of the time (SD ± 5.0%). As additional research is conducted, this analysis can serve as a baseline against which other metrics can be tested, and moving forward, these benchmarks can be utilized in situations where referees are equal in other categories.
This research revealed two additional findings that were not part of the core hypothesis for this project. The first is that referees in men's and women's hockey make correct penalty calls at the same rate. This illustrates that, despite the differences in rules and style of play, the evaluation of a referee's judgement does not change between men's and women's hockey. The second is that the evaluators had a very low rate of agreement on what constituted a "marginal" penalty call. Evaluators tended to agree unanimously on what constituted an on-standard or marginal penalty call, but only agreed on 35% of marginal penalty calls. This indicates that this may not be a useful category for evaluating a referee's judgement and warrants further investigation.
Comments