An optimized framework for processing multicentric polysomnographic data incorporating expert human oversight

Benedikt Holm; Gabriel Jouan; Emil Hardarson; Sigríður Sigurðardottir; Kenan Hoelke; Conor Murphy; Erna Sif Arnardóttir; María Óskarsdóttir; Anna Sigríður Islind

doi:10.3389/fninf.2024.1379932

An optimized framework for processing multicentric polysomnographic data incorporating expert human oversight

Front Neuroinform. 2024 May 13:18:1379932. doi: 10.3389/fninf.2024.1379932. eCollection 2024.

Authors

Benedikt Holm^{1

2}, Gabriel Jouan^{1

2}, Emil Hardarson^{1

2}, Sigríður Sigurðardottir², Kenan Hoelke^{2

3}, Conor Murphy^{2

4}, Erna Sif Arnardóttir^{1

2}, María Óskarsdóttir^{1

2}, Anna Sigríður Islind^{1

2}

Affiliations

¹ Department of Computer Science, Reykjavik University, Reykjavik, Iceland.
² School of Technology, Reykjavik University Sleep Institute, Reykjavik, Iceland.
³ Board of Registered Polysomnographic Technologists, Arlington, VA, United States.
⁴ Physical Activity, Physical Education, Sport and Health Research Centre (PAPESH), Sports Science Department, School of Social Sciences, Reykjavik University, Reykjavik, Iceland.

Abstract

Introduction: Polysomnographic recordings are essential for diagnosing many sleep disorders, yet their detailed analysis presents considerable challenges. With the rise of machine learning methodologies, researchers have created various algorithms to automatically score and extract clinically relevant features from polysomnography, but less research has been devoted to how exactly the algorithms should be incorporated into the workflow of sleep technologists. This paper presents a sophisticated data collection platform developed under the Sleep Revolution project, to harness polysomnographic data from multiple European centers.

Methods: A tripartite platform is presented: a user-friendly web platform for uploading three-night polysomnographic recordings, a dedicated splitter that segments these into individual one-night recordings, and an advanced processor that enhances the one-night polysomnography with contemporary automatic scoring algorithms. The platform is evaluated using real-life data and human scorers, whereby scoring time, accuracy, and trust are quantified. Additionally, the scorers were interviewed about their trust in the platform, along with the impact of its integration into their workflow.

Results: We found that incorporating AI into the workflow of sleep technologists both decreased the time to score by up to 65 min and increased the agreement between technologists by as much as 0.17 κ.

Discussion: We conclude that while the inclusion of AI into the workflow of sleep technologists can have a positive impact in terms of speed and agreement, there is a need for trust in the algorithms.

Keywords: agreement; explainable AI; human-in-the-loop; machine learning; platform; scoring time; sleep research; trust.

Grants and funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. The authors of this paper have received funding to craft this paper from the European Union's Horizon 2020 research and innovation programme (grant agreement 965417) as well as NordForsk (NordSleep project 90458) via Business Finland (5133/31/2018), the Icelandic Research Fund (ESA & ASI). The Sleep Revolution project has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement No. 965417. The sleep technologists who scored the studies—Heidur Grétarsdóttir, Marjo Sunnari, Beate Diecker, Jacob Siegert, Dina Fernandes, Cátia Lígia Rito de Oliveira, Elena Robbi, Paul Murphy, and Alexander Ryan—are especially thanked for there essential contribution to this paper, as well as Kristín Anna Ólasfdóttir, who led the team of scorers.