Skip to main content

Mobility Vision

Mobility Vision focuses on multi-sensor perception and vision-based understanding, enabling robots to interpret their environment more comprehensively.

By combining multiple visual inputs and perception algorithms, robots can simultaneously perform depth estimation, scene understanding, and object recognition.

This section introduces the core concepts of multi-sensor vision systems, along with example implementations provided in the Robotic Suite.

1. System Overview​

A Mobility Vision system typically integrates:

  • Sensors: Multiple 3D cameras (RGB-D / Stereo)
  • Perception:
    • Depth estimation
    • Object detection
    • Semantic segmentation
  • Fusion: Multi-camera perception pipeline
  • Planning & Action: Navigation and robot control

This corresponds to a perception-driven pipeline:

  • /camera + /depth (multi)(or + /scan) → perception → /plan → /cmd_vel

2. Key Capabilities​

  • 3D Scene Reconstruction
    Build real-time volumetric maps using depth data (nvblox)

  • Object Detection (AI)
    Detect and classify objects using deep learning models (YOLOv8)

  • Semantic Segmentation
    Understand scene structure at pixel level (UNet)

  • Multi-Sensor Fusion
    Combine geometry and AI perception into a unified representation

  • Real-Time Perception Pipeline
    Run multiple perception modules simultaneously

3. Demo & Sample Code​

The Robotic Suite provides sample applications demonstrating multi-sensor vision-based perception.

Available Samples​

  • 3.2 Multi-Sensor Function (Multi-Camera + AI)
    Demonstrates multiple perception modules running together using:

    • Dual 3D camera inputs
    • Object detection
    • Semantic segmentation
    • 3D mapping

    Current implementation uses two 3D cameras as input for multi-sensor perception.

These samples are all packaged in containers to enable rapid evaluation and learning.

👉 Please refer to the corresponding sample code pages for:

  • Step-by-step setup instructions
  • Launch commands
  • Code structure and customization

4. Use Cases​

  • Intelligent service robots
  • Autonomous inspection systems
  • Smart retail and human-aware navigation
  • Advanced robotics research (AI + perception fusion)

5. Considerations​

  • Requires high computational resources (CPU/GPU)
  • System integration complexity is higher
  • Synchronization between multiple perception modules is critical
  • Performance tuning (latency, throughput) is important

Mobility Vision represents the evolution from single-sensor systems to integrated, AI-powered robotic perception, enabling more intelligent and adaptive robot behavior.

Next Step: Explore Vendor Samples & Ecosystems to see how industry platforms accelerate robotic development.