HumanCOIN adds gaze, EMG and EEG proxies to 12,500-hour dataset
HumanCOIN organizes 12,500 hours of first-person manipulation data into attention, exertion and cognition tiers, aiming to expose signals that precede visible actions while leaving model-performance gains unproven.
Researchers behind HumanCOIN say they released a 12,500-hour first-person manipulation dataset that pairs visible activity with proxies for attention, physical exertion and cognition. The paper presenting HumanCOIN, submitted to arXiv on November 19, describes recordings across 38 scene classes and divides them into three progressively richer sensor tiers.
The authors say conventional observation-action pairs leave the intent between what a person sees and does unobserved. HumanCOIN adds synchronized signals for where an operator is likely looking, how their muscles are activating and what scalp electrical activity accompanies task decisions. Those signals could serve as conditioning data or auxiliary training targets before an action becomes fully visible. The release materials, however, do not show that they improve action prediction, task completion, generalization or another benchmark.
The largest tier, L1, contains 10,000 hours and infers gaze from headband-camera video. The authors present that attention proxy as a way to identify an operator’s next object of attention or reach before the hand moves. It is not directly measured eye tracking, and the paper does not establish the inference model’s accuracy, timing lead or failure modes.
L2 adds surface electromyography, or EMG, from armbands to 2,000 hours of recordings while retaining the L1 attention signal. The authors treat the electrical activity associated with muscle activation as a proxy for expected physical load before an action is complete. In principle, that extra signal could help distinguish visually similar movements that require different effort. The paper specifies eight electrodes per forearm sampled at 2,000 Hz and a fixed processing chain, but it does not demonstrate that the proxy generalizes across operators.
The 500-hour L3 tier adds scalp electroencephalography, or EEG, and inherits both lower-tier proxy modalities. The authors characterize EEG as a cognition proxy for early task decisions and monitoring, not as a direct readout of intent. In its controlled setting, L3 also directly measures gaze, hand pose and finger kinematics. The combined streams could let researchers align an emerging task decision with later attention, muscle activation and movement. The available materials do not establish whether an EEG-conditioned model would generalize across participants or outside the collection setup.
Across the tiers, HumanCOIN reports multi-view RGB images, depth maps, point clouds, six-degree-of-freedom head and wrist poses, and step-level action and object annotations. The stated tier durations—10,000, 2,000 and 500 hours—sum to the reported 12,500-hour total. The paper says clock and calibration data are included to realign sensors collected at different rates.
The official HumanCOIN toolkit stores camera, EEG, EMG, inertial, pose and action streams independently in time-indexed Parquet data. Instead of forcing every source onto one frame rate, it uses configurable time-window queries and includes alignment checks. Its documentation points to repositories on Hugging Face and ModelScope; the HumanCOIN ModelScope organization listed one L3 repository when checked.
The release leaves important questions unresolved. The paper does not report a participant count or demographics, a formal ethics review, predefined train-validation-test splits or how much of the full corpus is currently downloadable. It does document written consent, withdrawal rights, de-identification and face blurring, human review of annotations, and technical quality checks. The paper assigns CC BY-NC 4.0 to the data and Apache-2.0 to the code, while the observed ModelScope dataset listing carried an Apache-2.0 label, so prospective users should resolve the applicable repository terms before reuse.
More news

EngineAI introduces Awaken-HI for humanoid training

PhyDEX paper puts three VLA models below 3% success

Researchers train VLA robot policies to stop safely mid-task
