For individuals with Profound Intellectual Multiple Disabilities (PIMD), the inability to verbally communicate their needs creates a critical gap that isolates patients and exhausts caregivers. The Center for Research Computing (CRC) is addressing this challenge through VOICE4PIMD, a project that combines artificial intelligence, advanced sensor integration, multicloud and the CRC’s on-premise computing infrastructure to interpret non-verbal communication patterns and translate them into actionable insights.
The VOICE4PIMD system captures multiple data streams, including facial expressions, heart rate, eye gaze, audio vocalizations, seizure events, and environmental factors, then uses multimodal AI to predict how an individual might be feeling or what they need. Critically, the models are trained to understand each patient's unique, idiosyncratic communication patterns, creating a personalized interpretation system that functions as a universal translator for caregiving.
Under the leadership of CRC Director Jarek Nabrzyski and Research Software Engineer Evan Brinckman, and through the work of summer interns Joon Kim (Purdue), Kristin Kelly, Layann Wardeh, Elliot Kim, Alphonsus Koong Bok Hui (all from ND), the Center provided essential research computing expertise that proved critical to the project's viability. The team tackled a fundamental deployment challenge: consolidating powerful AI models onto a compact edge AI device suitable for real-world home use. This hardware integration effort combined embedded sensors for temperature, humidity, video, and audio capture with a night-vision camera and speaker, making sophisticated edge computing practical for continuous patient monitoring.
Equally crucial was the data infrastructure developed by the team. An automated data acquisition pipeline that streams annotated sensor data to the cloud or to on-premises resources for model training and personalization was built. The architecture leverages AWS cloud functions for transcription and processing, enabling continuous model refinement as the system learns each individual's communication patterns. The project's technical architecture has potential for broad application. The same approach to interpreting physiological data and non-verbal cues through personalized AI extends beyond PIMD to patients with Alzheimer's, dementia, stroke rehabilitation, and other conditions limiting verbal communication.
The VOICE4PIMD project demonstrates how the Center's expertise can translate an initial idea into cutting-edge technology with real-world impact.
