Back to the previous page
nature_communications
Monday 10 December 2018 18:30

Medical informatics: A scientific publication in the field of medical image analysis has been accepted in Nature Communications. It is the result of the research work of Prof. Henning Müller of the Institute of Information Systems HES-SO Valais-Wallis.

Publication

“Why rankings of biomedical image analysis competitions should be interpreted with care“

Authors

Lena Maier-Hein, Matthias Eisenmann, Annika Reinke, Sinan Onogur, Marko Stankovic, Patrick Scholz, Tal Arbel, Hrvoje Bogunovic, Andrew Bradley, Aaron Carass, Carolin Feldmann, Alejandro Frangi, Peter Full, Bram van Ginneken, Allan Hanbury, Katrin Honauer, Michal Kozubek, Bennett Landman, Keno März, Oskar Maier, Klaus Maier-Hein, Bjoern Menze, Henning Müller, Peter Neher, Wiro Niessen, Nasir Rajpoot, Gregory Sharp, Korsuk Sirinukunwattana, Stefanie Speidel, Christian Stock, Danail Stoyanov, Abdel Aziz Taha, Fons van der Sommen, Ching-Wei Wang, Marc-André Weber, Guoyan Zheng, Pierre Jannin, and Annette Kopp-Schneide

Abstract

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.