Real-time assistance in suicide prevention helplines using a deep learning-based recommender system: A randomized controlled trial

Salim Salmi; Saskia Mérelle; Nikki van Eijk; Renske Gilissen; Rob van der Mei; Sandjai Bhulai

doi:10.1016/j.ijmedinf.2024.105760

Real-time assistance in suicide prevention helplines using a deep learning-based recommender system: A randomized controlled trial

Int J Med Inform. 2024 Dec 17:195:105760. doi: 10.1016/j.ijmedinf.2024.105760. Online ahead of print.

Authors

Salim Salmi¹, Saskia Mérelle², Nikki van Eijk², Renske Gilissen², Rob van der Mei³, Sandjai Bhulai⁴

Affiliations

¹ Research, 113 Suicide Prevention, Paasheuvelweg 25, 1105 BP Amsterdam-Zuidoost, Netherlands; Stochastics, Centrum Wiskunde & Informatica, P.O. Box 94079, 1090 GB Amsterdam, Netherlands; Mathematics, Faculty of Science, Vrije Universiteit Amsterdam, De Boelelaan 1105, 1081 HV Amsterdam, Netherlands. Electronic address: s.salmi@113.nl.
² Research, 113 Suicide Prevention, Paasheuvelweg 25, 1105 BP Amsterdam-Zuidoost, Netherlands.
³ Stochastics, Centrum Wiskunde & Informatica, P.O. Box 94079, 1090 GB Amsterdam, Netherlands.
⁴ Mathematics, Faculty of Science, Vrije Universiteit Amsterdam, De Boelelaan 1105, 1081 HV Amsterdam, Netherlands.

PMID: 39705915
DOI: 10.1016/j.ijmedinf.2024.105760

Abstract

Objective: To evaluate the effectiveness and usability of an AI-assisted tool in providing real-time assistance to counselors during suicide prevention helpline conversations.

Methods: In this RCT, the intervention group used an AI-assisted tool, which generated suggestions based on sentence embeddings (i.e. BERT) from previous successful counseling sessions. Cosine similarity was used to present the top 5 chat situation to the counsellors. The control group did not have access to the tool (care as usual). Both groups completed a questionnaire assessing their self-efficacy at the end of each shift. Counselors' usage of the tool was evaluated by measuring frequency, duration and content of interactions.

Results: In total, 48 counselors participated in the experiment: 27 counselors in the experimental condition and 21 counselors in the control condition. Together they rated 188 shifts. No significant difference in self-efficacy was observed between the two groups (p=0.36). However, counselors that used the AI-assisted tool had marginally lower response time and used the tool more often during conversations that had a longer duration. A deeper analysis of usage showed that the tool was frequently used in inappropriate situations, e.g. after the counselor had already provided a response to the help-seeker, defeating the purpose of the information. When the tool was employed appropriately (64 conversations), it provided usable information in 53 conversations (83%). However, counselors used the tool less frequently at optimal moments, indicating their potential lack of proficiency with using AI-assisted tools during helpline conversations or initial trust issues with the system.

Conclusion: The study demonstrates benefits and pitfalls of integrating AI-assisted tools in suicide prevention for improving counselor support. Despite the lack of significant impact on self-efficacy, the support tool provided usable suggestions and the frequent use during long conversations suggests counsellors may wish to use the tool in complex or challenging interactions.

Keywords: AI; Helpline; Machine learning; RCT; Support tool.