Multimodal machine learning framework for driver attention estimation in conditionally automated driving

Peña Arias, Juan C.

Publication

Multimodal machine learning framework for driver attention estimation in conditionally automated driving

Peña Arias, Juan C.

Abstract

Autonomous driving technologies have the potential to improve road safety, comfort, and accessibility. In particular, conditional automation allows the vehicle to perform the driving task under specific conditions, but still requires the human driver to remain available to resume control when the system reaches its operational limits or when a takeover request is issued. This requirement makes continuous driver attention estimation an important component of safe human–automation interaction. In this context, this thesis developed and evaluated a multimodal driver attention estimation system based on deep learning, integrating non-invasive physiological signals from wearable sensors (ECG and EDA), hands-on-wheel behavioral data, and surrounding traffic-context variables. To support this objective, an experimental framework was implemented using a motion platform and an automated driving environment developed in CARLA. Data were collected from 30 participants, and attention levels were labeled using a previously validated secondary visual detection task. The synchronized multimodal data were then used to train and evaluate an early-fusion LSTM model through leave-one-subject-out cross-validation and Bayesian hyperparameter optimization. Results showed that the multimodal approach improved attention estimation compared with physiology-only and other reduced-modality configurations, achieving high classification performance and demonstrating the complementary value of behavioral and traffic-context information. These findings support the use of multimodal driver monitoring systems to assess attention in conditional automated driving.
Las tecnologías de conducción automatizada pueden mejorar la seguridad vial, el confort y la accesibilidad. En particular, la automatización condicional permite que el vehículo ejecute la tarea de conducción bajo ciertas condiciones, pero aun requiere que el conductor esté disponible para retomar el control cuando el sistema alcanza sus límites operacionales o emite una solicitud de toma de control. Por ello, la estimación continua de la atención del conductor es importante para una interacción segura entre el humano y la automatización. En este contexto, esta tesis desarrolló y avaluó un sistema multimodal de estimación de la atención basado en aprendizaje profundo, integrando señales fisiológicas no invasivas obtenidas con sensores vestibles (ECG y EDA), información conductual de manos en el volante y variables del tráfico circundante. Para ello, se implementó un marco experimental con una plataforma de movimiento y un entorno de conducción automatizada desarrollado en CARLA. Se recolectaron datos de 30 participantes y los niveles de atención se etiquetaron mediante una tarea visual secundaria.