A story of a challenge and how to keep it impartial

In the biomedical image processing field, new algorithms for segmentation, detection, classification are introduced continuously. The comparison of the proposed solution with the existed ones is crucial to show the importance of the new methods. Therefore challenges are getting more prevalent and important than ever before. However, there are still big flaws in their designs and implementations. This talk will start by presenting a brief history of the field and will address these flaws as well as possible methods and suggestions in order to keep challenges fair. A small demonstration will also be presented, showing how single metrics may fail to evaluate the overall quality of a solution. To further elaborate on the topic, the CHAOS challenge (Combined CT-MR Healthy Abdominal Organ Segmentation) will be introduced. Finally, the experiences of a challenge organizer will be shared with tips and advice.

• A story of a challenge and how to keep it impartial (PDF | 3,45 MB)