GEAR-SONIC Introduction#

SONIC is a humanoid behavior foundation model that gives robots a core set of motor skills learned from large-scale human motion data. Rather than building a separate controller for each predefined motion, SONIC treats motion tracking as a scalable training task, enabling a single unified policy to produce natural, whole-body movement and to support a wide range of behaviors — from walking and crawling to teleoperation and multi-modal control.

This repository implements a comprehensive optimization of the SONIC whole-body-control (WBC) inference pipeline on the Intel Core Ultra “Panther Lake” (PTL) platform, including OpenVINO inference acceleration, a real-time control thread design, and priority-based NPU scheduling. It demonstrates that the PTL platform can meet SONIC WBC’s determinism requirements while achieving substantial power savings compared to GPU execution.

Prerequisites#

  • Follow the Get Started guide to set up the base system.

  • NPU driver (> 1.32.0)

  • Ubuntu 24.04 RT release

  • OpenVINO 2026.3, ROS2 Jazzy

For OpenVINO, this project recommends installing from the archive file. Follow Install OpenVINO from an Archive File (Linux), download the latest OpenVINO package, and extract it to the /opt/intel/<openvino_version> folder, e.g. /opt/intel/openvino_2026.3.0.

Installation#

This project extends the open-source GR00T-WholeBodyControl project to add OpenVINO acceleration and SONIC WBC pipeline optimizations for the Intel Core Ultra Panther Lake platform. Set up the environment with the following steps.

1. Initialize and patch the submodule#

git submodule update --init <GR00T-WholeBodyControl>
cd <GR00T-WholeBodyControl>

2. Apply the patches#

git am --whitespace=fix ../patches/*.patch

3. Model preparation#

python3 download_from_hf.py

This downloads the default deployment models — planner, encoder, and decoder — into gear_sonic_deploy/.

Tip: If the download fails or stalls due to a proxy/network issue reaching Hugging Face, try switching to a mirror endpoint before re-running the script:

export HF_ENDPOINT="https://hf-mirror.com"

4. Build#

Set up the build environment and build the project:

source gear_sonic_deploy/scripts/setup_env.sh
cd gear_sonic_deploy
just build

setup_env.sh auto-detects your OpenVINO installation and sets USE_OPENVINO=1, so the build links against OpenVINO instead of CUDA/TensorRT.

Basic Tests: Run SONIC with MuJoCo#

One-time setup#

From the repo root, install the MuJoCo simulator environment:

bash install_scripts/install_mujoco_sim.sh --ov

This creates a lightweight .venv_sim virtual environment with only the packages needed for the simulator (MuJoCo, Unitree SDK2, etc.).

Tip: Suggest run the MuJoCo simulator on a local display, not over a remote connection (SSH X11 forwarding, remote desktop, etc.). Remote rendering adds its own latency/jitter, which can make a correct motion (walking, running, dancing) look wrong.

Run the basic SONIC test#

Terminal 1 — MuJoCo simulator:

source .venv_sim/bin/activate
python gear_sonic/scripts/run_sim_loop.py

Terminal 2 — Deployment:

cd gear_sonic_deploy
bash deploy.sh sim

Starting control#

  1. In Terminal 2 (deploy.sh), press ] to start the policy.

  2. Click on the MuJoCo viewer window, then press 9 to drop the robot to the ground.

  3. Go back to Terminal 2. Press T to play the current reference motion — the robot executes it to completion.

  4. Press N or P to switch to the next or previous motion sequence. Press T again to play the new motion.

  5. When you are done, or need an emergency stop, press O to stop control and exit.

  6. Press Enter to toggle planner mode.

  7. In planner mode, use N or P to switch between motion modes (IDLE, SQUAT, BOXING, LEDGE_WALKING, etc.).

  8. In IDLE or LEDGE_WALKING mode, use the keys below to control the robot.

Planner mode keys (toggle planner mode with Enter):

Key

Action

W / S

Move forward / backward

A / D

Adjust heading slightly and move forward (left / right)

Q / E

Turn in place — adjust facing direction only, no forward motion (±30° per press)

1–8

Select a locomotion mode from the current motion set

N / P

Next / previous motion set

9 / 0

Decrease / increase speed

T

Play motion

For the full control reference — including Normal Mode, all motion sets, and the momentum system — see docs/source/tutorials/keyboard.md.

Checking Model Performance#

Once control is running, the deploy log periodically prints a Loop timing line with running latency stats (in microseconds) for the control loop:

Loop timing - ... Obs: 645us (avg:689 min:460 P99:907 max:1023), Policy: 405us (avg:405 min:306 P99:618 max:723), ...

  • Obs — gathering observations for the current tick plus running the encoder.

  • Policy — pure decoder inference time.

In planner mode, an added Planner block reports the planner model’s own latency the same way, e.g. Model: 15403us (avg:15304 min:14895 P99:16292 max:17895).

Use these numbers to confirm real-time deadlines are met and to compare latency across NPU priority/turbo settings (see NPU Inference Configuration below).

NPU Inference Configuration#

The inference: section of gear_sonic_deploy/policy/release/observation_config.yaml controls how the encoder, policy, and planner models are scheduled on the NPU:

  • NPU priority (encoder_priority / policy_priority / planner_priority): HIGH, NORMAL, or LOW. Controls how the OpenVINO scheduler arbitrates models that share the same NPU device (mapped to ov::hint::model_priority). Encoder and policy default to HIGH since they run on the real-time control thread; the planner defaults to NORMAL.

  • NPU turbo mode (npu_turbo): when true, applies NPU_TURBO=YES to all NPU models, trading power efficiency for lower inference latency. Recommended on for real-time control; set to false to save power/thermal headroom. This is an NPU-only hint and is ignored on CPU/GPU.

Verifying the Inference Device#

To confirm which device a model landed on, watch the deploy log during model init (encoder, policy, planner) — before Starting control is printed. Each NPU model logs:

[OVInference] Requested device: NPU
[OVInference] Model priority: HIGH

Encoder and policy should show HIGH priority; the planner should show NORMAL.