Hecate-Omni

Hecate-Omni: State-Aligned Hierarchical Credit Assignment for Multimodal Social Reasoning

Hecate-Omni Overview

Contributions in Hecate-Omni

1. We develop and release Hecate-Omni, a model for multimodal social intelligence reasoning, designed to handle diverse social behavior tasks spanning emotion recognition, mental-state analysis, sentiment understanding, and pathological behavior recognition. Hecate-Omni achieves SOTA on the evaluated benchmarks, demonstrating effective and robust reasoning across different social behavior scenarios.

2. We introduce Hecate-PO. Hecate-PO adaptively identifies latent reasoning units using predictive entropy and establishes correspondences among equivalent reasoning states through internal state representations, enabling reasoning decisions to be evaluated under semantically comparable contexts.

Experimental Results

Results

Training detail and evaluation

Evaluation

MODEL_PATH=/path/to/base_model \
DATA_PATH=/path/to/human_behavior_atlas \
ADAPTER=/path/to/checkpoint \
EVAL_SPLIT=test EVAL_PER_DATASET=0 RUN_NAME=hecatepo_test \
bash scripts/run_hecatepo.sh evaluate

Prepare

Download the base model OmniSapiens2.0 and Qwen2.5-Omni-7B from huggingface

Prepare Hecate-Omni LoRA weights from huggingface

Prepare human-behavior-atlas data from [huggingface](https://huggingface.co/datasets/Harry-1234/Hecate-Omni

Training

MODEL_PATH=/path/to/base_model \
DATA_PATH=/path/to/human_behavior_atlas \
RUN_NAME=hecatepo_train \
bash scripts/run_hecatepo.sh train

Examples

https://github.com/user-attachments/assets/000a936d-2834-4798-88f5-3d7d52c58610 ``` Question: How would you describe the emotional state of the speaker in the recording? Choose the most appropriate emotion from: anger, disgust, fear, happy, surprise, neutral, sad

First reason briefly from the provided evidence. Enclose your reasoning in .... Then provide exactly one final answer in \boxed{...}. For classification, use exactly one of the requested labels. For open questions, give a concise, direct answer.