cua-s1-forms-onnx

ONNX conversion of cua-ai/cua-s1-forms, prepared for lightweight local inference experiments with ONNX Runtime, ONNX Runtime Web, WebAssembly (WASM), and WebGPU.

This repository is a format conversion / deployment adaptation of the original CUA-S1 Forms checkpoint.
The model architecture, training work, checkpoint, and original research belong to the CUA-S1 project and its authors.
This repository does not claim to retrain, improve, or reproduce the original model.

Original model

• Original Hugging Face model: https://huggingface.co/cua-ai/cua-s1-forms
• Original project/source: https://github.com/trycua/cua/tree/main/libs/cua-s1
• Converted repository: https://huggingface.co/yasserrmd/cua-s1-forms-onnx
• Original license: MIT
• Conversion: PyTorch checkpoint → ONNX
• Target runtimes: ONNX Runtime CPU, ONNX Runtime Web/WASM, ONNX Runtime Web/WebGPU

Please read the original model card before using this conversion. It contains the authoritative description of the model's architecture, training setup, evaluation, limitations, and intended scope.

What is CUA-S1 Forms?

CUA-S1 Forms is a small specialist "System One" model for form-oriented computer-use tasks.

Unlike an autoregressive LLM, it does not generate text token-by-token. For each UI element, it receives:

1. a short structured context describing the task, form, and element, and
2. a set of candidate options such as extracted document values plus actions like check, click, and skip.

It returns one score per candidate option in a single forward pass.

The original model card describes the architecture as:

• byte-level input representation
• 2-layer Transformer encoder
• width 128
• 4 attention heads
• option-attention scoring head
• approximately 706K trainable parameters
• approximately 2.8 MB original checkpoint

This makes the model especially interesting for local, low-latency decision workloads where running a large generative model for every UI action would be unnecessary.

Why this ONNX conversion?

The original Hugging Face repository currently publishes a PyTorch checkpoint:

cua-s1-forms.pt

This repository converts that trained checkpoint into ONNX so the same decision model can be tested outside a Python/PyTorch runtime.

The primary goal is to explore a deployment path like:

Structured UI state
        |
        v
Candidate actions / values
        |
        v
   CUA-S1 ONNX
        |
        v
  Option logits
        |
        v
   Select action

For browser experiments:

Browser application
        |
        v
UTF-8 byte encoding
        |
        v
ONNX Runtime Web
   /           \
 WASM         WebGPU
   \           /
        |
        v
 CUA-S1 Forms
        |
        v
 Local decision

No server-side inference is required by the model itself.

Files

ONNX input contract

The browser-oriented export uses fixed token dimensions and dynamic batch / candidate dimensions.

Inputs

context_ids
int64 [batch, context_tokens]

context_mask
bool  [batch, context_tokens]

option_ids
int64 [batch, options, option_tokens]

option_token_mask
bool  [batch, options, option_tokens]

option_mask
bool  [batch, options]

The configuration used by the model defines:

context_tokens = 224
option_tokens  = 96

Therefore the practical tensor shapes are:

context_ids:
[B, 224]

context_mask:
[B, 224]

option_ids:
[B, N, 96]

option_token_mask:
[B, N, 96]

option_mask:
[B, N]

where:

• B = batch size / number of UI elements scored together
• N = number of candidate options

Output

logits
float32 [batch, options]

Apply softmax across the option dimension if probabilities are required.

Byte encoding

CUA-S1 uses a deliberately simple byte-level representation.

For a UTF-8 encoded string:

byte_id = byte + 1

0 is reserved for padding.

Conceptually:

def encode_bytes(text, max_length):
    raw = text.encode("utf-8")[:max_length]

    ids = [b + 1 for b in raw]
    ids += [0] * (max_length - len(ids))

    return ids

The corresponding mask is True for real bytes and False for padding.

This means a browser implementation does not require a tokenizer library. JavaScript's TextEncoder is sufficient.

Example context

A context follows the original CUA-S1 task structure, for example:

TASK fill the form from the document, then submit
FORM Northwind Clinic - New Patient Registration
ELEMENT Edit "Phone number" value=""

Candidate options might look like:

fill Name: John Smith
fill Email: [email protected]
fill Tel: (503) 555-0142
fill DOB: 03/14/1987
fill State: Oregon
check
click
skip

The model returns one logit for each option.

Python ONNX Runtime example

import numpy as np
import onnxruntime as ort

session = ort.InferenceSession(
    "cua-s1-forms.onnx",
    providers=["CPUExecutionProvider"],
)

outputs = session.run(
    ["logits"],
    {
        "context_ids": context_ids,
        "context_mask": context_mask,
        "option_ids": option_ids,
        "option_token_mask": option_token_mask,
        "option_mask": option_mask,
    },
)

logits = outputs[0]

The tensors must follow the shapes and dtypes documented above.

Browser direction

This conversion was created primarily to evaluate CUA-S1 as a fully client-side decision model.

The intended browser stack is:

TypeScript / JavaScript
        |
        v
ONNX Runtime Web
        |
   +----+----+
   |         |
 WebGPU     WASM
   |         |
   +----+----+
        |
        v
cua-s1-forms.onnx

A Web Worker is recommended so model loading and inference do not block the UI thread.

Important

The ONNX model has been validated with ONNX Runtime CPU.

WebGPU and WASM browser performance should be measured independently. For a model this small, WebGPU is not automatically guaranteed to outperform WASM/CPU because GPU dispatch overhead can dominate very small workloads.

Initial ONNX Runtime CPU benchmark

A preliminary benchmark was run in Google Colab using ONNX Runtime CPUExecutionProvider.

Configuration:

Batch size:       1
Candidate options: 8
Runs:             2000

Observed latency:

These are preliminary environment-specific measurements, not official CUA-S1 benchmark results.

Colab is a shared execution environment, so tail latency can include runtime scheduling noise. Browser WASM and WebGPU benchmarks will be reported separately after validation.

Conversion validation

The conversion process checks:

Original trained PyTorch checkpoint
        |
        v
Reconstructed CUA-S1 architecture
        |
        v
ONNX export
        |
        +--> ONNX checker
        |
        +--> ONNX Runtime inference
        |
        +--> PyTorch vs ONNX numerical parity
        |
        +--> dynamic batch validation

The export uses fixed context and option token lengths because the PyTorch Transformer ONNX graph can otherwise embed traced sequence lengths inside attention reshape operations.

Thus:

context length       = fixed
option token length  = fixed

batch size           = dynamic
candidate count      = dynamic

This is also a practical inference contract for browser runtimes.

What this repository changes

This repository changes only the deployment representation.

It does not change:

• model architecture
• learned weights
• training data
• training procedure
• model behavior by design
• original evaluation results

The primary changes are:

• PyTorch checkpoint converted to ONNX
• fixed token dimensions for stable ONNX execution
• browser-oriented metadata added
• runtime compatibility and performance testing added

Intended use

This conversion is intended for:

• research
• evaluation
• local inference experiments
• browser inference experiments
• form-oriented computer-use research
• testing small specialist decision models
• comparing CPU, WASM, and WebGPU execution

It is particularly useful for exploring architectures where a small decision model acts as a fast action-selection layer rather than invoking a large generative model for every UI element.

Limitations

The limitations of the original model continue to apply.

According to the original model card, important limitations include:

• It is a specialist form-oriented model, not a general-purpose computer-use agent.
• It selects among provided candidate values/actions rather than inventing arbitrary values.
• The original training is based on synthetic form episodes, with a relatively small real-world evaluation set.
• The byte-level vocabulary and training setup are English-centric.
• Performance should not be assumed to transfer to arbitrary forms, applications, operating systems, languages, layouts, or accessibility configurations.
• Model output should not itself be treated as proof that an external action succeeded.

This ONNX conversion does not remove or mitigate those limitations.

Safety

Computer-use systems should separate decision from execution.

A suitable architecture is:

Observe
   |
   v
Build bounded action space
   |
   v
CUA-S1 decision
   |
   v
Validate
   |
   v
Execute
   |
   v
Verify outcome

For demonstrations, it is preferable to restrict execution to a sandboxed DOM or controlled test environment.

For consequential, irreversible, financial, legal, medical, account, permission, or external-communication actions, human review and independent outcome verification should be required.

See the original CUA-S1 project documentation for the authoritative safety guidance.

Attribution

This repository is derived from:

CUA-S1 Forms

Original model:

https://huggingface.co/cua-ai/cua-s1-forms

Original source:

https://github.com/trycua/cua/tree/main/libs/cua-s1

The original model is described as a small, specialist "System One" option scorer for form-oriented computer-use research and is designed to work as a decision layer behind CUA Driver.

ONNX conversion

Converted and tested for ONNX / browser deployment experiments by:

Mohamed Yasser

Hugging Face:

https://huggingface.co/yasserrmd

This conversion is independent deployment work and is not an official CUA-AI release.

Citation

If you use the underlying model or discuss its architecture/results, please cite and reference the original CUA-S1 project rather than this conversion alone:

CUA-S1 project:
https://github.com/trycua/cua/tree/main/libs/cua-s1

Original model:
https://huggingface.co/cua-ai/cua-s1-forms

If you specifically use this ONNX conversion, you may additionally reference:

https://huggingface.co/yasserrmd/cua-s1-forms-onnx

License

The original cua-s1-forms Hugging Face model is published under the MIT License. This converted model is distributed under the same license.

Please review the original model repository and upstream source for the authoritative license text and third-party notices.

Status

Current:

PyTorch checkpoint loading      PASS
ONNX export                     PASS
ONNX validation                 PASS
PyTorch / ONNX parity           PASS
ONNX Runtime CPU inference      PASS
Dynamic batch execution         PASS
CPU performance testing         IN PROGRESS / VALIDATED
Browser WASM testing            NEXT
Browser WebGPU testing          NEXT

The next stage is a fully client-side Hugging Face Space demonstrating the same ONNX model with ONNX Runtime Web using both WASM and WebGPU execution providers.