AI can detect serious neurologic changes in babies in the NICU using video data alone

Findings from this new artificial intelligence (AI)-based tool, published November 11 in Lancet’s eClinicalMedicine, could lead to a minimally invasive, scalable method for continuous neurologic monitoring in NICUs, providing critical real-time insights into infant health that have not been possible before.

Every year, more than 300,000 newborns are admitted to NICUs across the United States. Infant alertness is considered the most sensitive piece of the neurologic exam, reflecting integrity throughout the central nervous system. Neurologic deterioration in NICUs can happen unexpectedly and has devastating consequences. However, unlike cardiorespiratory telemetry, which continuously monitors the heart and lung function of babies in the NICU, neurotelemetry has remained elusive in most NICUs despite decades of work in electroencephalography (EEG) and specialized neuro-NICUs. Neurologic status is evaluated intermittently, using physical exams that are imprecise and may miss subacute changes.

The Mount Sinai team hypothesized that a computer vision method to track infant movement could predict neurologic changes in the NICU. “Pose AI” is a machine learning method that tracks anatomic landmarks from video data; it has revolutionized athletics and robotics.

The Mount Sinai team trained an AI algorithm on more than 16,938,000 seconds of video footage from a diverse group of 115 infants in the NICU at The Mount Sinai Hospital undergoing continuous video EEG monitoring. They demonstrated that Pose AI can accurately track infant landmarks from video data. They then used anatomic landmarks from the video data to predict two critical conditions — sedation and cerebral dysfunction — with high accuracy.

“Although many neonatal intensive care units contain video cameras, to date they do not apply deep learning to monitor patients,” said Felix Richter, MD, PhD, senior author of the paper and Instructor of Newborn Medicine in the Department of Pediatrics at Mount Sinai. “Our study shows that applying an AI algorithm to cameras that continuously monitor infants in the NICU is an effective way to detect neurologic changes early, potentially allowing for faster interventions and better outcomes.”

The research team was surprised by how well Pose AI worked across different lighting conditions (day vs. night vs. in babies receiving phototherapy) and from different angles. They were also surprised that their Pose AI movement index was associated with both gestational age and postnatal age.

“It’s important to note that this approach does not replace the physician and nursing assessments that are critical in the NICU. Rather, it augments these by providing a continuous readout that can then be acted on in a given clinical context,” explained Dr. Richter. “We envision a future system where cameras continuously monitor infants in the NICU, with AI providing a neuro-telemetry strip similar to heart rate or respiratory monitoring, with alert for changes in sedation levels or cerebral dysfunction. Clinicians could review videos and AI-generated insights when needed, offering an intuitive and easily interpretable tool for bedside care.”

The team noted the limitations of the study, including that the AI models were trained on data collected at a single institution, meaning that this algorithm and neurologic predictions need to be evaluated on video data from other institutions and video cameras. The research team plans to test this technology in additional NICUs and to develop clinical trials that will assess its impact on care. They are also exploring its application to other neurological conditions and expanding its use to adult populations.

“At Mount Sinai, we are committed to ensuring that new artificial intelligence possibilities are investigated and leveraged to advance care for our patients,” said Girish N. Nadkarni MD, MPH, System Chief of Data Driven and Digital Medicine, Director of the Mount Sinai Clinical Intelligence Center, Director of The Charles Bronfman Institute for Personalized Medicine and a study co-author. “AI tools are already advancing clinical care across the Mount Sinai Health System, including by shortening length of stay, reducing hospital readmissions, aiding in cancer diagnostics and therapeutic targeting, and delivering real-time care to patients based on physiological data generated from wearables, to name a few. We are excited to now be bringing this non-invasive, safe, and effective AI tool into the NICU to improve outcomes for our smallest, most fragile patients.”