Open Science

Main contribution: We make openly accessible the first video-rate spectral imaging database comprising over 200,000 images from ten different subjects and different levels of perfusion.

Spectral imaging has the potential to become a key technique in interventional medicine as it unveils much richer optical information compared to conventional RBG (red, green, and blue)-based imaging. Thus allowing for high-resolution functional tissue analysis in real time. Its higher information density particularly shows promise for the development of powerful perfusion monitoring methods for clinical use. However, even though in vivo validation of such methods is crucial for their clinical translation, the biomedical field suffers from a lack of publicly available datasets for this purpose. Closing this gap, we generated the SPECTRAL Perfusion Arm Clamping dAtaset (SPECTRALPACA). It comprises ten spectral videos (∼20 Hz, approx. 20,000 frames each) systematically recorded of the hands of ten healthy human participants in different functional states. We paired each spectral video with concisely tracked regions of interest, and corresponding diffuse reflectance measurements recorded with a spectrometer. Providing the first openly accessible in human spectral video dataset for perfusion monitoring, our work facilitates the development and validation of new functional imaging methods.

Ayala L., Mindroc-Filimon D., Rees M., Hübner M., Sellner J., Seidlitz S., Tizabi M., Wirkert S., Seitel A. & Maier-Hein L. (2024) The SPECTRAL Perfusion Arm Clamping dAtaset (SPECTRALPACA) for video-rate functional imaging of the skin. Nature Scientific Data 11. [pdf]

Main contributions: First biomedical image analysis competition involving a diverse range of tasks and modalities, which demonstrated that segmentation algorithms performing well on multiple tasks in the challenge could effectively generalize to previously unseen tasks and potentially outperform custom-designed solutions.

IInternational challenges have become the de facto standard for comparative assessment of image analysis algorithms. Although segmentation is the most widely investigated medical image processing task, the various challenges have been organized to focus only on specific clinical tasks. We organized the Medical Segmentation Decathlon (MSD) - a biomedical image analysis challenge, in which algorithms compete in a multitude of both tasks and modalities to investigate the hypothesis that a method capable of performing well on multiple tasks will generalize well to a previously unseen task and potentially outperform a custom-designed solution. MSD results confirmed this hypothesis, moreover, MSD winner continued generalizing well to a wide range of other clinical problems for the next two years. Three main conclusions can be drawn from this study: (1) state-of-the-art image segmentation algorithms generalize well when retrained on unseen tasks; (2) consistent algorithmic performance across multiple tasks is a strong surrogate of algorithmic generalizability; (3) the training of accurate AI segmentation models is now commoditized to scientists that are not versed in AI model training.

Antonelli, M., Reinke, A., Bakas, S., Farahani, K., Kopp-Schneider, A., Landman, B. A., ... , Maier-Hein, L., Cardoso, M. J. (2022). The medical segmentation decathlon. Nature communications, 13(1), 4128. [pdf]

Main contribution: Open-source toolkit for simulation and image processing of optical and acoustic imaging.

Significance: Optical and acoustic imaging techniques enable noninvasive visualisation of structural and functional properties of tissue. The quantification of measurements, however, remains challenging due to the inverse problems that must be solved. Emerging data-driven approaches are promising, but they rely heavily on the presence of high-quality simulations across a range of wavelengths due to the lack of ground truth knowledge of tissue acoustical and optical properties in realistic settings.
Aim: To facilitate this process, we present the open-source simulation and image processing for photonics and acoustics (SIMPA) Python toolkit. SIMPA is being developed according to modern software design standards.
Approach: SIMPA enables the use of computational forward models, data processing algorithms, and digital device twins to simulate realistic images within a single pipeline. SIMPA's module implementations can be seamlessly exchanged as SIMPA abstracts from the concrete implementation of each forward model and builds the simulation pipeline in a modular fashion. Furthermore, SIMPA provides comprehensive libraries of biological structures, such as vessels, as well as optical and acoustic properties and other functionalities for the generation of realistic tissue models.
Results: To showcase the capabilities of SIMPA, we show examples in the context of photoacoustic imaging: the diversity of creatable tissue models, the customisability of a simulation pipeline, and the degree of realism of the simulations.
Conclusions: SIMPA is an open-source toolkit that can be used to simulate optical and acoustic imaging modalities. The code is available at: https://github.com/IMSY-DKFZ/simpa , and all of the examples and experiments in this paper can be reproduced using the code available at: https://github.com/IMSY-DKFZ/simpa_paper_experiments.

Gröhl, J., Dreher, K. K., Schellenberg, M., Rix, T., Holzwarth, N., Vieten, P., Ayala, L., Bohndiek, S. E., Seitel, A., & Maier-Hein, L. (2022). SIMPA: An open-source toolkit for simulation and image processing for photonics and acoustics. Journal of Biomedical Optics, 27(8), 083010. [pdf]

Main contributions: Introduction of the Heidelberg Colorectal (HeiCo) data set, the first publicly available data set designed to comprehensively benchmark medical instrument detection and segmentation algorithms, with a specific focus on method robustness and generalization capabilities.

Image-based tracking of medical instruments is an integral part of surgical data science applications. Previous research has addressed the tasks of detecting, segmenting and tracking medical instruments based on laparoscopic video data. However, the proposed methods still tend to fail when applied to challenging images and do not generalize well to data they have not been trained on. This paper introduces the Heidelberg Colorectal (HeiCo) data set - the first publicly available data set enabling comprehensive benchmarking of medical instrument detection and segmentation algorithms with a specific emphasis on method robustness and generalization capabilities. Our data set comprises 30 laparoscopic videos and corresponding sensor data from medical devices in the operating room for three different types of laparoscopic surgery. Annotations include surgical phase labels for all video frames as well as information on instrument presence and corresponding instance-wise segmentation masks for surgical instruments (if any) in more than 10,000 individual frames. The data has successfully been used to organize international competitions within the Endoscopic Vision Challenges 2017 and 2019.

Maier-Hein, L., Wagner, M., Ross, T., Reinke, A., Bodenstedt, S., Full, P. M., ... & Müller-Stich, B. P. (2021). Heidelberg colorectal data set for surgical data science in the sensor operating room. Scientific data, 8(1), 101. [pdf]

Main contributions: First large collection of annotated medical image datasets for semantic segmentation covering various clinically relevant anatomies, made available under an open-source license.

Semantic segmentation of medical images aims to associate a pixel with a label in a medical image without human initialization. The success of semantic segmentation algorithms is contingent on the availability of high-quality imaging data with corresponding labels provided by experts. We sought to create a large collection of annotated medical image datasets of various clinically relevant anatomies available under open source license to facilitate the development of semantic segmentation algorithms. Such a resource would allow: 1) objective assessment of general-purpose segmentation methods through comprehensive benchmarking and 2) open and free access to medical image data for any researcher interested in the problem domain. Through a multi-institutional effort, we generated a large, curated dataset representative of several highly variable segmentation tasks that was used in a crowd-sourced challenge - the Medical Segmentation Decathlon held during the 2018 Medical Image Computing and Computer Aided Interventions Conference in Granada, Spain. Here, we describe these ten labeled image datasets so that these data may be effectively reused by the research community.

Simpson, A. L., Antonelli, M., Bakas, S., Bilello, M., Farahani, K., Van Ginneken, B., ... , Maier-Hein, L., Cardoso, M. J. (2019). A large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv preprint arXiv:1902.09063. [pdf]