Conference on Robot Learning (CoRL) 2026

CoSMO: Conformal Semantic Mapping
for Open-World Safe Robot Navigation

Shuqi Wang^1,2Yiwei Zheng¹Pingrui Zhang³Yue Gao^1,2Xiang Yin^1,†

¹ Shanghai Jiao Tong University² Shanghai Innovation Institute³ Fudan University

† Corresponding author

Navigation paths in an indoor scene with known and unseen semantic hazards: geometry-only, VLM-only and CoSMO. — CoSMO constructs calibrated semantic coverage sets and uses category-dependent clearances to navigate around known and newly discovered hazards.

Abstract

Safe autonomous navigation requires not only geometric obstacle avoidance but also semantic awareness. Different object categories demand different safety clearances, and the set of relevant categories may expand as the robot explores novel environments. Vision-language models (VLMs) offer rich semantic representations for building maps of unknown scenes, yet their confidence scores are poorly calibrated, making raw predictions unreliable for safety-critical planning. Conformal prediction (CP) can provide coverage guarantees, but existing methods assume a fixed label set and stationary data, limiting their use in open-world settings. We propose Conformal Semantic Mapping for Open-World Safe Robot Navigation (CoSMO), a semantic safety mapping framework that transforms VLM embeddings into calibrated coverage sets and safety-aware obstacle dilations over a dynamically expanding hazard vocabulary, even under distribution shift. Experiments on the Matterport3D (MP3D) dataset show that CoSMO reduces semantic collisions by 61.6% relative to a geometry-only baseline, while maintaining formal coverage guarantees even under delayed expert feedback. Real-world experiments on a Unitree Go2 robot further demonstrate generalization beyond simulation, achieving long-run empirical coverage of 90.73% against the 90% target under delayed detector feedback.

TL;DR: CoSMO converts vision-language embeddings into calibrated semantic safety maps for robot navigation with formal safety guarantees. It adapts online to distribution shifts, including the addition of new hazard categories, without additional offline training or calibration.

Method

CoSMO adds a semantic safety layer to vision-based navigation. Dense RGB-D features are projected into an embedding map, conformal coverage sets represent plausible labels, and obstacles are dilated according to the largest required safety clearance in each set. The downstream planner then consumes this safety-aware obstacle map.

CoSMO framework: RGB-D observations, VLM encoder, embedding map, calibrated coverage sets, semantic dilation, path planning and delayed expert feedback. — Overview of CoSMO. Delayed expert feedback expands the safety dictionary and updates the adaptive conformal threshold online.

Embedding-space calibration

Nonconformity is measured in the shared vision-language embedding space. Adding new labels preserves the compatibility of previously collected calibration scores.

Online expert feedback

Adaptive conformal inference tracks changing observations. Delayed expert labels update the calibration threshold and reveal new hazard categories.

Semantic safety constraints

The robot plans against the largest safety distance among the labels in each coverage set, accounting for uncertainty rather than selecting only one label.

We prove that CoSMO reaches its target coverage in the long run even when expert feedback arrives late. We also bound the risk of the robot getting too close to obstacles.

Simulation Results

We evaluate on the full Matterport3D validation split of 2,195 ObjectNav episodes. Starting with an empty hazard dictionary, open-vocabulary CoSMO reduces semantic collision rate from 25.5% to 9.8% relative to geometry-only OneMap, a 61.6% relative reduction.

Navigation results on all 2,195 MP3D validation episodes
Method	SR ↑	SCR ↓	SPL ↑
OneMap geometry only	11.3%	25.5%	0.071
GCLIP Argmax	18.4%	13.1%	0.103
Uniform Dilation	8.7%	8.9%	0.053
CoSMO open vocabulary	13.8%	9.8%	0.081
CoSMO known vocabulary	10.4%	9.4%	0.059
CoSMO Grounding DINO expert	16.5%	10.5%	0.097

SR: success rate. SCR: semantic collision rate. SPL: success weighted by path length. Safety clearances introduce a safety–success tradeoff in narrow indoor layouts.

Safety and success Pareto frontier for CoSMO and baselines. — Safety–success tradeoff on 330 ablation episodes. CoSMO dominates offline CP across the evaluated coverage targets.

Adaptive threshold and coverage trajectories with different expert feedback delays. — Delay ablation on 330 episodes. Across tested delays of 0–100 steps, SCR remains within 10.0–10.9%.

Real-World Experiments

We deploy CoSMO on a Unitree Go2 robot in two indoor scenes. The robot searches for a chair while avoiding household objects and free-form semantic concepts, using GCLIP for perception and Grounding DINO as the delayed expert oracle.

Watch on YouTube ↗

Across 30 real-world trials, CoSMO achieves 90.73% empirical coverage against a 90% target, a 16.67% semantic collision rate, and a 46.67% navigation success rate. Measured expert feedback delays range from 128.1 to 447.6 ms.

Unitree Go2 navigation, semantic maps and trajectories in a real indoor experiment. — Real-world navigation in an initially unknown 3.5 m × 10 m indoor environment with GCLIP perception and delayed Grounding DINO feedback.

Citation

The paper link will be added when the preprint is available.

@inproceedings{wang2026cosmo,
  title     = {CoSMO: Conformal Semantic Mapping
               for Open-World Safe Robot Navigation},
  author    = {Wang, Shuqi and Zheng, Yiwei and
               Zhang, Pingrui and Gao, Yue and Yin, Xiang},
  booktitle = {Conference on Robot Learning},
  year      = {2026}
}

Preliminary citation; final publication metadata will be updated.