Overview
Metrics are reported in the paper under LOSO cross-validation and sensor-dropout scenarios (S0–S6). System-level validation uses Kinova Gen3 hardware-in-the-loop (HIL) evaluation of supervisor variants.
Abstract
Safe commit-to-execute decisions are critical for assistive robotic manipulators: the system must minimize false activations during rest while remaining responsive to user intent. Existing intent-recognition pipelines conflate window-level classification with commit readiness, yielding high false-positive rates and instability under cross-subject variability and sensor dropout. We present NeuroCommitSSM, a decision-centric framework that learns continuous commit-readiness from synchronized electroencephalography (EEG), electromyography (EMG), and eye tracking (ET) using neurophysiology-aware encoders and entropy-regularized multimodal fusion. Discrete commit events are generated via dwell/hysteresis filtering and coupled with a HOLD–ASSIST–COMMIT (HAC) supervisor that gates execution on perception and robot-state feasibility. Under leave-one-subject-out (LOSO) cross-validation (N=32, five International Classification of Functioning, Disability and Health (ICF)-aligned activities of daily living (ADL) tasks), NeuroCommitSSM achieves 0.950 action balanced accuracy and 0.75 false commits per 1000 s of rest, with graceful degradation under sensor dropout. Hardware-in-the-loop (HIL) validation on a Kinova Gen3 arm confirms safe execution under real-world constraints.
Method
Commit Readiness from EEG–EMG–ET (Model Architecture)
NeuroCommitSSM operates on synchronized tri-modal windows of length 2.0 s (T=500 samples at 250 Hz; stride 0.25 s) and propagates per-sample validity masks throughout tokenization, encoding, fusion, and pooling. For each window, the model jointly predicts (i) REST vs. ACTION, (ii) task class (T1–T5) for ACTION windows, and (iii) a continuous commit-readiness score ct∈[0,1] that is converted to stable commit events by dwell/hysteresis filtering and consumed by the HOLD–ASSIST–COMMIT supervisor.
- Masked patch tokenization: masked patch projection (p=25 → L=20) into D=160 embeddings; patch validity suppresses unreliable tokens.
- Encoders: EEG multiscale filterbank + virtual-channel mixing (K=6); EMG envelope/burst pathways + synergy basis (M=4); ET validity-aware salience with top-k event tokens.
- Reliability-aware fusion: quality proxies → uncertainty weights; entropy regularization discourages single-modality dominance.
- Auxiliary features: compact descriptors (EEG PSD + EMG time-domain + modality flags; 123-D) injected for dropout robustness.
- Heads: action (REST/ACTION), task (T1–T5), and commit head predicting continuous readiness used by dwell/hysteresis filtering.
Pipeline
Pipeline and HAC Shared Autonomy (Data Collection → Evaluation)
The system implements an end-to-end intent-to-commit workflow that converts synchronized EEG–EMG–ET into window-level predictions and commit-to-execute decisions. Commit readiness is stabilized via dwell/hysteresis filtering and integrated into a ROS2 supervisor (HOLD–ASSIST–COMMIT) that gates execution on perception and robot-state feasibility for safe, repeatable behavior under cross-subject shift and sensor dropout.
- Data acquisition: synchronized EEG (8ch @ 250 Hz), EMG (4ch), and eye tracking via iMotions/LSL during five ICF-aligned ADL tasks plus REST.
- Preprocessing & windowing: deterministic filtering/repair per modality; ET validity handling; 2.0 s windows with 0.25 s stride; train-only normalization.
- Commit formation: readiness ct → discrete commits using dwell + hysteresis (separate ON/OFF thresholds; optional cooldown) to suppress transient spikes.
- Feasibility gating: perception feasibility fcv(t) + robot feasibility frobot(t) (IK, collisions, joint limits). Safety aborts return to HOLD.
- Evaluation: LOSO + sensor-dropout scenarios (S0–S6) with decision-quality metrics (FP/1k REST, flaps/min, coverage) and Kinova Gen3 HIL supervisor ablations.
Video
This page includes (i) the data-collection protocol clips for each task (T1–T5) and (ii) a Kinova Gen3 HAC shared-autonomy demonstration video. If a video does not load due to hosting/browser limitations, use the “Open” button beneath the clip to view it on GitHub.
Kinova Gen3 Demonstration (HAC Shared Autonomy)
Project Demo — HAC on Kinova Gen3
End-to-end execution under feasibility-grounded HOLD–ASSIST–COMMIT supervision with commit-to-execute decision gating.
Task Videos (Data Collection Protocol)
The following clips illustrate the data-collection protocol for each task (T1–T5), including participant interaction with the setup and the trial workflow.
Results
The paper reports results for (i) action and task recognition under LOSO cross-validation, (ii) commit decision quality under sensor-dropout scenarios (S0–S6), and (iii) system-level performance with Kinova Gen3 HIL supervisor ablations. Metrics include action/task balanced accuracy and AUPRC, plus decision-quality metrics (FP/1k REST, flaps/min, and success-to-commit coverage), with paired statistical tests and ablations.
Dataset
We release a synchronized tri-modal dataset consisting of EEG (Enobio-8, 8 channels @ 250 Hz), EMG (PLUX, 4 channels), and eye tracking (Pupil Labs Neon) recorded via iMotions/LSL during five ICF-aligned ADL tasks plus REST. Evaluation follows subject-independent leave-one-subject-out (LOSO) splits and sensor-dropout scenarios (S0–S6).
- T1 (Clock): reaching toward a tabletop clock and grasping it.
- T2 (Bottle): picking up a dropped apple juice bottle and placing it back onto the table.
- T3 (Fan): pressing the fan power button to toggle it on and off.
- T4 (Plant): adjusting or repositioning a plant on the shelf.
- T5 (Wave): performing a greeting gesture by waving the hand.
BibTeX
If you use this work, please cite:
@inproceedings{sultan2026neurocommitssm,
title = {NeuroCommitSSM: Decision-Centric Shared Autonomy for Safe Assistive Manipulation via EEG--EMG--ET Commit Readiness},
author = {Sultan, Tipu and Sangani, Param and Cool, Kody and Sikorski, Pascal and Liu, Guangping and Akbarpour, Hadi and Babaiasl, Madi},
booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
year = {2026},
note = {Under review}
}