OpenSLA

Sensor-Language-Action Models

News

• [2026-10-06] Our paper is available on arXiv.
• [2026-10-06] Code released on GitHub, and model released on HuggingFace.
• [2026-10-06] Project website is live!

Introduction

OpenSLA is a new family of models - Sensor-Langauge-Action (SLA) models. It takes a multi-channel sensor history together with language context and produces a structured action prediction and a sensor-state caption. We release two variants:

• OpenSLA-B: sensor tokens are projected and prepended to the language prompt as a flat token prefix.
• OpenSLA-H: builds on B with a hierarchical sensor encoder that compresses the signal into local, per-channel, and global memory tokens.

Table of Contents

1. Installation
2. Quick Start
3. Pretrained Weights
4. Usage
5. Datasets
6. Citation

Installation

git clone https://github.com/yang-ai-lab/Sensor-Language-Action-Model.git
cd Sensor-Language-Action-Model
pip install -r requirements.txt

Dependencies

• Python >= 3.10
• PyTorch >= 2.5
• Transformers >= 5.8.1 (for the Qwen3.5 backbone)
• PEFT >= 0.12

Quick Start

demo.ipynb loads a checkpoint, builds an input batch, and runs prediction. The same flow in Python:

import torch
from opensla import OpenSLA, SensorBatch

model = OpenSLA.from_checkpoint(
    "pretrained_weights/opensla_h_clinical.pt", domain="clinical",
    dino_checkpoint="pretrained_weights/waveform_encoder.ckpt",
    numeric_config="pretrained_weights/numeric_config.json",
    action_group_types="pretrained_weights/action_group_types.json",
)

batch = torch.load("data/preprocessed_batch.pt", weights_only=True)
predictions = model.predict(batch["input_text"], SensorBatch(**batch["sensors"]))

Pretrained Weights

Download the files from this repository and put them under pretrained_weights/ in the code repository.

Usage

Input Format

The model takes preprocessed signals as a .pt dictionary with domain, input_text (one text context per sample), and sensors. The sensors are:

• Waveform: waveform of shape [B, M, C, T], with M one-minute slots, C physical channels, and T = 7500 samples per slot (60 s at 125 Hz), plus a boolean waveform_channel_mask ([B, M, C]) marking observed channel-minutes and int64 waveform_modality_ids ([B, C]) indexing opensla.WAVEFORM_MODALITIES[domain].
• Numeric: observed events as parallel [B, E] tensors (values, rel_time_min in minutes before the decision, measure_ids, source_ids, event_mask), plus per-measure summary features (summary_features, summary_measure_ids).

Command line

Point configs/template.json at your weights and input batch, then:

opensla --config configs/template.json --dry-run   # print the resolved config
opensla --config configs/template.json             # write outputs/predictions.jsonl

On Slurm:

sbatch --account=YOUR_ACCOUNT --partition=YOUR_PARTITION scripts/run_template.sbatch --config configs/template.json

Datasets

The models are trained and evaluated on six datasets from three healthcare settings, with an additional MIMIC-IV held out as an external clinical cohort. All of them are publicly available and can be accessed through the following links.

Citation

If you use this code or models in your research, please cite our paper:

@misc{xu2026opensla,
  title  = {Sensor-Language-Action Models},
  author = {Xu, Yuekai and Shuai, Zitao and Yang, Yuzhe},
  journal = {arXiv preprint},
  year = {2026}
}

Acknowledgments

The waveform encoder adapts from the sensor encoder from OSF (MIT License).