KinHEC

KinHEC: Kinematic Trajectory Benchmark for Hand-Eye and Robot-World Calibration

Toolkit on GitHub | pip install kinhec | MIT license

KinHEC is a large-scale, fully synthetic, ground-truth-complete benchmark for the two classical sensor-calibration problems of robotics,

• hand-eye calibration, A X = X B, and
• simultaneous robot-world / hand-eye calibration, A X = Y B,

together with their spatio-temporal extension (unknown clock offset between the robot and the camera). Instead of images, every sequence consists of two time-stamped pose streams, the robot flange trajectory (from forward kinematics of real manipulator models) and the pose of the observed reference in the camera, both as exact ground truth and as measurements corrupted by physically motivated noise models. This isolates the numerical calibration problem from image processing so that solvers can be compared under precisely controlled conditions: sensor noise, motion degeneracy, temporal misalignment, outliers, unknown monocular scale, systematic robot model errors and image-formation (PnP) realism.

The smoke/small tiers are meant for testing; the full tier (77,628 sequences, more than 500 factor levels across eight tracks) is the benchmark proper. All tiers come from the same deterministic generator, so any copy can be regenerated or extended with the commands in Regenerating and extending.

Quick start

pip install kinhec huggingface_hub
hf download Ezharjan/KinHEC --repo-type dataset --local-dir kinhec_data

from kinhec import load_manifest, load_sequence, solvers, se3

man = load_manifest("kinhec_data")                       # one row per sequence, every factor as a column
seq = load_sequence("kinhec_data", "noise-000000", man)  # both streams and the ground truth
A, B = seq.axxb_motions()                                # relative motions between keyframes
X = solvers.park_martin(A, B)                            # solve A X = X B
print(se3.pose_error(X, seq.X))                          # rotation error [deg], translation error [mm]

--include restricts the download to what a study needs, which matters because the whole copy is 50.1 GB:

hf download Ezharjan/KinHEC --repo-type dataset --local-dir kinhec_data --include "sequences.parquet" "robot/noise-*.parquet" "camera/noise-*.parquet" "robots.json" "cameras_targets.json" "generation_info.json"

Why another calibration dataset?

Hand-eye calibration papers usually report results on a few hundred simulated pose pairs with isotropic Gaussian noise plus one or two in-house recordings. Real recordings (e.g. the ETH ASL hand-eye datasets of Furrer et al. 2017, or the image sets of Koide and Menegatti 2019) are valuable but small, tied to one robot/camera pair and without ground truth for the calibration itself. The consequences are well documented (Ali et al. 2019; Enebuse et al. 2021): rankings of the classical solvers change from paper to paper, robustness claims are hard to verify, and learning-based solvers lack training data. KinHEC addresses this with

1. ground truth for everything: the hand-eye transform X, the robot-world transform Y, the true clock offset, the monocular scale, per-frame outlier and visibility labels, the true (perturbed) DH parameters of "uncalibrated" robots, and the exact robot pose at every camera instant;
2. kinematic realism: 13 published manipulator models (Universal Robots CB3/e-Series, Franka Emika Panda, KUKA LBR iiwa 7/14, ABB IRB 120, PUMA 560) with joint limits and speed caps, smooth joint-space excitation, move-and-dwell station captures, and IK-feasible "look-at" poses that keep the target in the field of view of a pinhole camera;
3. systematic factor sweeps (eight tracks) that give robustness curves instead of single numbers, including a controllable degeneracy dial (cone half-angle of the rotation axes) and systematic robot-model errors that no i.i.d. noise model reproduces;
4. both configurations (eye-in-hand and eye-to-hand) and both formulations (AX=XB on relative motions, AX=YB on absolute poses) from the same files;
5. a reference toolkit with eleven baseline solvers (Tsai-Lenz, Park-Martin, Horaud-Dornaika, Daniilidis, Andreff, Shah, Li, non-linear and RANSAC variants, time-offset estimation), a fixed evaluation protocol and a validation tool, so that numbers are comparable across papers.

Tracks

Sequences per robot in this copy:

Configurations: eye_in_hand: 47,778, eye_to_hand: 29,850. Trajectory families: cartesian_cone: 3,728, joint_sinusoid: 28,802, joint_waypoints: 19,640, lookat: 24,516, proposal_sine: 942. Camera noise models: pnp: 7,827, se3_gaussian: 54,729, vo_drift: 15,072.

Named noise levels

The level string of a sequence names the factors its track sweeps: noise names the two noise levels (robot=r2|camera=c3), while the other systematic tracks hold the noise fixed and name their own factors instead (cone=10|rot=8-15|n=20, offset=0.05|rate=30|jitter=0.002|drop=0.1, pixel_std=0.5|camera=hd|target=...). Whatever the level string says, the numeric values are in the per-sequence columns robot_rot_std_rad, robot_trans_std_m, joint_std_rad, dh_len_std_m, dh_ang_std_rad, camera_rot_std_rad, camera_trans_std_m, camera_trans_std_z_m and pixel_std, each of which is 0 or NaN where the corresponding model is not active (the robot_error track, for instance, uses joint_gaussian and dh_error, so its robot_rot_std_rad and robot_trans_std_m are 0).

Robot levels (flange-frame twist noise, standard deviations):

Camera levels (target orientation noise; translation noise in the camera frame, lateral x/y and along the optical axis z):

(Values taken from configs/full.yaml, the tier definition that produced this copy. A tier names every level it might use; the levels that a given track actually sweeps are the ones that appear in its level strings, so the noise-free r0 and c0 are defined here but unused in the systematic tracks.)

Coordinate frames and conventions

• T_a_b is the 4x4 homogeneous pose of frame b expressed in frame a (p_a = T_a_b p_b). In the files a pose is stored as px, py, pz [m] and a unit quaternion qw, qx, qy, qz with qw >= 0.
• Frames: base (robot base), ee (robot flange), cam (camera optical frame, OpenCV convention: x right, y down, z forward), target (the observed reference, see below), world (fixed frame in which Y is expressed).
• Robot stream (robot/*.parquet): time stamps on the robot clock (the reference clock), joint positions (true, and as read from the encoders) and the flange pose T_base_ee (ground truth and reported/measured).
• Camera stream (camera/*.parquet): time stamps on the camera clock, and T_cam_target, the pose of the observed reference in the camera frame (what a PnP / marker detector or a VO system actually measures), ground truth and measured. The reference is
• the static calibration board (world_frame = target, eye-in-hand),
• the marker mounted on the flange, seen by a static camera (world_frame = target, eye-to-hand), or
• the origin of the visual-odometry frame, i.e. the first camera pose (world_frame = vo_origin, monocular sequences).

world_frame therefore distinguishes a fixed physical reference from a VO origin, not eye-in-hand from eye-to-hand; the configuration column does that. * Ground truth (sequences.parquet): X_* and Y_* as position + quaternion.

Relative motions for A X = X B follow from two pairs i, j: A = inv(A_i) A_j, B = inv(B_i) B_j (the toolkit method Sequence.axxb_motions() builds them with keyframe selection).

Time: a camera frame captured at true (robot-clock) time t_gt is stamped t = t_gt + time_offset_s + jitter; the column t_gt and the ground-truth flange pose at t_gt are stored in the camera table so that the temporal and the spatial parts of the problem can be evaluated separately. Synchronous sequences have identical clocks and frame-aligned rows in the two tables.

Files

README.md  LICENSE  CITATION.cff
sequences.parquet, sequences.csv     index: one row per sequence (ground truth, factor levels, file locations)
robot/<track>-<shard>.parquet        robot stream (one row group per sequence)
camera/<track>-<shard>.parquet       camera stream (one row group per sequence)
robots.json                          kinematic models (DH tables, limits, speed caps, sources)
cameras_targets.json                 camera intrinsics and calibration-target presets
generation_info.json, configs/       tier configuration that produced this copy
benchmarks/                          baseline results (results.parquet/csv, leaderboard.md, summary.json,
                                     protocol.json)
figures/                             overview figures

sequences.parquet (index)

One row per sequence with: identifiers (sequence_id, track, level, seed, generator_version), the setup (robot, dof, configuration, family, world_frame, camera, target), sampling (sampling_mode continuous|stations, sync, duration_s, robot_rate_hz, camera_rate_hz, n_robot_frames, n_camera_frames, n_visible_frames, n_outlier_frames), timing ground truth (time_offset_s, time_jitter_std_s, camera_drop_fraction), monocular scale and scale_drift_std, the noise factors (robot_noise_models, robot_rot_std_rad, robot_trans_std_m, joint_std_rad, dh_len_std_m, dh_ang_std_rad, camera_noise_model, camera_rot_std_rad, camera_trans_std_m, camera_trans_std_z_m, pixel_std, student_t_dof, outlier_fraction, outlier_model), the ground truth X_px..X_qz, Y_px..Y_qz, the motion descriptors (n_motions, axis_scatter_lambda2, axis_scatter_lambda3, mean_rot_angle_deg, max_rot_angle_deg, mean_trans_m, translation_cond), the file locations (robot_file, camera_file, *_row_group) and the complete generation recipe (config_json, extra_json: trajectory parameters, keyframe indices, true DH parameters, per-frame scale, camera intrinsics, target geometry). sequences.csv is the same table without the two JSON columns.

axis_scatter_lambda2 is the second eigenvalue of the scatter matrix of the rotation axes of the keyframe motions, each weighted by the square of its rotation angle (0 = all axes parallel = rotation of X unobservable; 1/3 = isotropic). translation_cond is the condition number of the stacked [R_A - I] blocks, infinite for pure translations; being a ratio of singular values it is scale invariant, so it describes the directional diversity of the rotation axes and should be read together with mean_rot_angle_deg, which carries the magnitude that decides how strongly measurement noise is amplified into t_X. Both rate columns are NaN for station-based sequences, whose two streams share the dwell instants.

robot stream columns

camera stream columns

Loading the data

The toolkit (pip install kinhec) reads a copy on disk and returns numpy arrays with the ground truth attached:

from kinhec import load_manifest, load_sequence, solvers, se3

man = load_manifest("kinhec_data")                                # pandas DataFrame, one row per sequence
pnp = man[(man.track == "pnp") & (man.pixel_std == 0.5)]          # filter on any factor column
seq = load_sequence("kinhec_data", pnp.sequence_id.iloc[0], man)
A_abs, B_abs = seq.axyb_poses()                                   # absolute pose pairs for A X = Y B
X, Y = solvers.shah(A_abs, B_abs)
print(se3.pose_error(X, seq.X), se3.pose_error(Y, seq.Y))

Plain PyArrow and pandas, without the toolkit (one row group per sequence, so a single sequence is read without touching the rest of the shard):

import pyarrow.parquet as pq

man = pq.read_table("kinhec_data/sequences.parquet").to_pandas()
row = man.iloc[0]
robot = pq.ParquetFile("kinhec_data/" + row.robot_file).read_row_group(int(row.robot_row_group)).to_pandas()
camera = pq.ParquetFile("kinhec_data/" + row.camera_file).read_row_group(int(row.camera_row_group)).to_pandas()

The Hugging Face datasets library, streaming straight from the Hub (one configuration per track and stream, as listed by the dataset viewer):

from datasets import load_dataset

index = load_dataset("Ezharjan/KinHEC", "sequences", split="train")
robot = load_dataset("Ezharjan/KinHEC", "robot_noise", split="train", streaming=True)
camera = load_dataset("Ezharjan/KinHEC", "camera_noise", split="train", streaming=True)

Manipulator models

reach [m] is the manufacturer's nominal working radius, measured to the wrist point; it is used only as a conservative bound when sampling reachable poses, and the flange can be a little further out at full extension. Joint zero conventions follow the manufacturers' tables (UR: official DH article; Panda: Franka Control Interface documentation, Craig's convention, flange 0.107 m beyond joint 7; iiwa: standard DH with the link lengths 0.34/0.40/0.40/0.126 m (iiwa 7) and 0.36/0.42/0.40/0.126 m (iiwa 14), cross-checked against the iiwa_stack URDF; IRB 120: the published 290/270/70/302/72 mm table with joint zero at the ABB calibration pose, tool0 at (0.374, 0, 0.630) m; PUMA 560: Corke's Robotics Toolbox model). The forward kinematics of the toolkit is checked against independent implementations (reference poses from the Robotics Toolbox for Python, a chain built from the iiwa_stack URDF) to better than 1e-9 m and 1e-9 rad. The joint-speed caps are nominal values used only to bound the generated motions (the generator uses at most 25 % of them, 35 % in the temporal track).

Camera and target presets

Visibility of a target requires all points (checkerboards) or at least the minimum number of points (grids / markers) to project inside the image with a 5 px margin, a depth between 0.1 m and 5 m and a viewing angle below 75 deg between the board normal and the line of sight. Visibility is computed for every sequence; for the pnp noise model a measurement exists only for visible frames, for the other noise models the flag is informative (the camera stream then represents a generic 6-DoF pose sensor). Sequences with world_frame = vo_origin have no board: visible is always true and n_points, depth_m, view_angle_deg are 0 / NaN.

Generation pipeline

For every sequence, with a private random generator seeded from seed:

1. Hand-eye transform X: camera (eye-in-hand) 2-12 cm in front of the flange, lateral offset up to 8 cm, optical axis within 40 deg of the flange z-axis, random roll; marker (eye-to-hand) 0-10 cm along the flange z-axis with a lateral offset up to 5 cm, normal within 30 deg of the flange z-axis.
2. Trajectory (continuous in time): * joint_sinusoid: every excited joint follows a sum of sinusoids (two harmonics by default, three with higher frequencies in the temporal track) around a configuration near the robot's home pose; amplitudes stay inside the joint limits and below 25-35 % of the joint-speed caps; optional joint subsets (single, parallel = two joints with parallel axes, two_axes, wrist, all), with non-excited joints held at the home configuration; * joint_waypoints: random joint-space waypoints connected by minimum-jerk segments with dwell times (move-stop-capture); station-based sampling records one frame per dwell; * lookat: observing poses (camera on a spherical cap in front of the board, or marker facing the static camera), converted to flange poses with the true X, Y and solved by damped-least-squares IK inside the joint limits; connected as joint_waypoints. For the free-floating free6d there is no chain to invert, so the observing poses are used directly and connected as Cartesian segments; * cartesian_cone (robot free6d): keyframe rotations R_k = R_{k-1} exp(theta_k a_k) whose axes a_k lie in a cone of given half-angle around a random axis (0 deg = single-axis rotations, fully degenerate); * proposal_sine (robot free6d): sinusoidal translation and Slerp between random keyframe rotations, the generator of the original KinHEC proposal document.
3. World reference Y: for lookat sampled first (board on a table in front of the robot, tilted towards it; static camera 0.9-1.6 m from the workspace centre); for the other families placed after the trajectory by drawing 25 candidate placements (board at the gaze point of a frame, facing the camera / static camera facing the marker) and keeping the one visible at the largest number of about 60 frames sampled evenly along the trajectory.
4. Clocks: synchronous sequences share one clock at the camera rate; asynchronous sequences have the robot stream at the controller rate (capped at 500 Hz) and camera frames with a random phase, clock offset, Gaussian jitter and random drops. Station sequences record the dwell instants only.
5. Exact poses of both streams and of T_cam_target from X, Y and the flange trajectory.
6. Robot measurement models (robot_noise_models): se3_gaussian - flange-frame twist noise T_meas = T exp(xi), xi ~ N(0, diag(rot_std^2 I, trans_std^2 I)); joint_gaussian - Gaussian noise on the joint readings propagated through the kinematics; dh_error - the true geometry differs from the nominal DH model by Gaussian errors on a, d (dh_len_std_m) and on alpha, theta_offset (dh_ang_std_rad); poses are reported with the nominal model, so the error is smooth, pose dependent and systematic. The perturbed parameters are stored in extra_json.true_dh.
7. Camera measurement models (camera_noise_model): se3_gaussian - translation noise in the camera frame with per-axis standard deviations (camera_trans_std_m lateral, camera_trans_std_z_m along the optical axis) and a right (target-frame) rotation perturbation with camera_rot_std_rad; pnp - the target points are projected with the true pose, Gaussian pixel noise pixel_std is added and the pose is re-estimated by Levenberg-Marquardt minimisation of the reprojection error (verified against OpenCV's iterative PnP), giving depth-dominated, pose-dependent, rotation/translation-correlated errors; vo_drift - noisy relative motions integrated into a random walk. The se3_gaussian and vo_drift models become multivariate Student-t when student_t_dof is set; the joint-encoder noise and the PnP pixel noise are always Gaussian, and in the shipped tiers only the camera stream ever uses heavy tails.
8. Monocular scale (scale > 1 or < 1): the camera translations in the VO frame are multiplied by scale (times a multiplicative random walk with scale_drift_std); the ground truth keeps the metric poses.
9. Outliers: a fraction outlier_fraction of the measurements is replaced (outlier flag) by a random pose (random_pose), by the mirrored solution of the planar pose ambiguity (planar_flip, board normal reflected about the line of sight) or by a gross Gaussian error (gross).
10. Descriptors of the keyframe motions (see the index columns) and packing into the tables.

Evaluation protocol and baselines

The protocol implemented in kinhec benchmark is identical for every solver: pair the streams (row-wise for synchronous sequences; for asynchronous sequences interpolate the measured robot stream at t - offset with the ground-truth offset, oracle, or with the offset estimated by angular-speed cross-correlation, estimated; that estimate comes with a confidence in time_offset_peak_corr, and the estimated results include the sequences where the correlation finds no peak, so filter on it before quoting a clock-offset accuracy), keep frames with a measurement, select keyframes at least 5 deg or 2 cm apart (at most 200; if fewer than three survive that filter, every paired frame is used), feed consecutive keyframe motions to A X = X B solvers and absolute keyframe poses to A X = Y B solvers, and report the geodesic rotation error [deg] and the Euclidean translation error [mm] of X (and Y).

succ. in the leaderboard counts the sequences for which a solver returned a finite estimate, which on the degenerate parts of the motion and outliers tracks it always does: a closed-form solver returns an orthonormal X whatever the data, so read the median errors, not the success rate, as the measure of whether a track was solved.

Median rotation error of X [deg] / median translation error of X [mm] per track, computed on 77,628 sequences of this copy with the ground-truth clock offset (oracle synchronisation). Full tables per solver, per factor level and with estimated time offsets: benchmarks/leaderboard.md; raw per-sequence results: benchmarks/results.parquet. andreff_scale, the eleventh baseline, applies only to the monocular sequences and is reported there rather than in this table.

Figures

The dataset

How one sequence is generated, from the ground-truth hand-eye transform to the two Parquet streams.

Composition of this copy, and the distribution of the motion descriptors that decide how well X is observable.

The manipulator models at their home configurations.

Sequence motion-002000 in the robot base frame: arm, flange path, camera frustums and the observed reference.

Sequence noise-000000 in the robot base frame: arm, flange path, camera frustums and the observed reference.

Sequence pnp-000180 in the robot base frame: arm, flange path, camera frustums and the observed reference.

The two streams of a sequence over time, ground truth and measurements, each on its own clock.

The measurement errors injected into that sequence, per component.

The pnp noise model in the image plane: target points projected with the true and with the measured pose.

Angular speed of the two streams of an asynchronous sequence, before and after clock-offset estimation.

Baseline results

Median translation error of X per track for the baseline solvers (oracle synchronisation).

Track noise: error against the camera noise level.

Track motion: error against the cone half-angle of the rotation axes, the degeneracy dial (0 deg leaves the rotation of X unobservable).

Track outliers: error against the outlier fraction, where the RANSAC baseline separates from the closed-form solvers.

Track temporal: error against the clock offset between the two streams.

Track robot_error: error against uncalibrated DH link-length errors.

Track pnp: error against pixel noise on the projected target points.

Calibration error against the motion descriptors of the individual sequences.

Regenerating and extending the dataset

This copy is not a recording: it is the output of a deterministic program, and the program is public.

pip install kinhec
kinhec plan --tier full                            # sequences per track
kinhec generate --tier full --out data_full --workers 8   # resumable; finished shards are skipped
kinhec validate data_full                          # integrity checks (see below)
kinhec benchmark data_full --workers 8             # baseline results into data_full/benchmarks
kinhec visualize data_full                         # the figures shown above
kinhec card data_full                              # refresh this README from the index
kinhec export data_full <sequence_id> --format csv # per-sequence CSV / TUM export

Every sequence is seeded from the tier's global seed and its own id, and shards are generated independently, so the result depends on neither the number of workers nor the order of execution. This copy was produced by kinhec 1.0.1, and the generator is unchanged in the current release, so the commands above regenerate the same sequences from the same seeds. Results are identical on a given platform and dependency set; across platforms and library versions the last few digits of a floating-point value can differ, as they do for any numerical pipeline.

Cost on the reference machine (a consumer desktop CPU, --workers 8): the full tier takes about 30 h to generate, roughly 11 s of single-core time per sequence on average, and the eleven baselines over all 77,628 sequences take about 7 h. The smaller tiers are minutes (smoke, 46 sequences; small, 562) to a few hours (medium, 9 996). A tier is a YAML file listing the factor levels of every track; copy one to build a custom tier (--tier my_tier.yaml), or raise repeats_multiplier to enlarge every track proportionally.

Validation

kinhec validate <data_dir> checks the presence of all files, the index against the shard row groups, the schemas, strictly increasing time stamps, orthonormal rotations and unit quaternions, the consistency of the frame/visibility/outlier counts, the temporal ground truth, that the forward kinematics of the true joint positions (with the true DH parameters of dh_error sequences) reproduces the ground-truth flange poses, and the exact identity A_i X = Y B_i on every camera frame of every sequence (tolerance 1e-7 deg / 1e-6 mm). The generated tiers ship validated.

Limitations

• Purely kinematic: no images, no dynamics/compliance, no calibration-target detection errors beyond the pixel-noise + PnP model and the planar-flip outlier model (an approximation of the two-fold ambiguity).
• The camera is an ideal pinhole with exactly known intrinsics: the pnp track models the geometry of image formation and its error structure, not lens distortion or the residual error of an intrinsic calibration.
• X and Y are exactly constant within a sequence; mounting flex and thermal drift are not modelled.
• Noise levels are chosen to bracket published sensor characteristics, not measured on a specific device.
• Self-collisions and workspace obstacles are not modelled; joint limits and speed caps are.
• The dh_error model perturbs all DH parameters independently; real geometric errors are correlated.

Citation

@misc{kinhec2026,
  title  = {KinHEC: A Kinematic Trajectory Benchmark for Hand-Eye and Robot-World Calibration},
  author = {Aizierjiang Aiersilan},
  year   = {2026},
  note   = {Dataset and toolkit},
  url    = {https://huggingface.co/datasets/Ezharjan/KinHEC}
}

Key references for the problem and for the implemented baselines: Shiu and Ahmad (1989); Tsai and Lenz (1989); Chou and Kamel (1991); Park and Martin (1994); Zhuang, Roth and Sudhakar (1994); Horaud and Dornaika (1995); Dornaika and Horaud (1998); Daniilidis (1999); Andreff, Horaud and Espiau (2001); Li, Wang and Wu (2010); Shah (2013); Tabb and Ahmad Yousef (2017); Furrer et al. (2017); Schweighofer and Pinz (2006) and Collins and Bartoli (2014) for the planar pose ambiguity. Full bibliographic details, including the sources of every kinematic model, are listed in the toolkit README.

License

MIT License, Copyright (c) 2026 Aizierjiang Aiersilan. See LICENSE.