Demo Video Explanation

This video demonstrates the complete sensor fusion pipeline in action. The system performs real-time object detection on camera images, then uses our calibrated transformation matrices to project the detected bounding boxes into the LiDAR coordinate system. All LiDAR points within each projected bounding box are colored with the same color as the corresponding camera detection, creating a unified multi-modal perception system.

Multi-Sensor Calibration and Sensor Fusion for Autonomous Vehicles

A comprehensive calibration system integrating cameras and LiDAR sensors for robust autonomous vehicle perception.

Technologies Used:
Python C++ ROS OpenCV scikit-learn NumPy RANSAC ICP SVD

Project Overview

As part of the Autonomous Vehicle Project at IIIT Delhi, I developed a comprehensive multi-sensor calibration system that solved one of the fundamental challenges in autonomous driving: precisely aligning data from multiple heterogeneous sensors to create a unified perception framework.

The system integrates 2 FLIR Pointgrey cameras and 3 VLP-16 LiDARs into a single, calibrated sensor suite with exceptional accuracy, enabling robust sensor fusion for autonomous navigation.

Key Achievements
  • <4° rotational error
  • <10cm translational error
  • 12+ FPS real-time performance
  • Multi-modal sensor fusion

The Technical Challenge

Autonomous vehicles rely on multiple sensors working in perfect harmony. However, each sensor has its own coordinate system, timing, and data format. Without precise calibration, sensor fusion becomes impossible, leading to inconsistent perception and dangerous navigation decisions.

Key Challenges:
  • Handle different sensor modalities (cameras vs LiDAR)
  • Achieve sub-degree rotational accuracy
  • Maintain centimeter-level translational precision
  • Work reliably in real-world conditions
  • Scale to multiple sensor configurations
System Requirements

The calibration system needed to handle:

  • 2 FLIR Pointgrey Cameras - Different viewpoints and intrinsics
  • 3 VLP-16 LiDAR sensors - Different mounting positions
  • Real-time constraints - Low latency for autonomous driving
  • Environmental robustness - Various lighting and weather conditions

My Technical Approach

1Data Acquisition and Synchronization Pipeline

I built the foundation using ROS (Robot Operating System) to create a synchronized data capture system. The key innovation was implementing precise temporal alignment between all sensors, ensuring that camera frames and LiDAR point clouds were captured at exactly the same timestamps.

Implementation Details:
  • Custom ROS nodes for each sensor type
  • Hardware-level synchronization triggers
  • Buffer management for different data rates
  • Real-time visualization in RViz for immediate feedback
Technical Innovation

Developed a custom synchronization protocol that handled the timing differences between 30Hz cameras and 10Hz LiDAR sensors, ensuring sub-millisecond temporal alignment.

2Intelligent Target Detection and Filtering

Rather than relying on simple threshold-based detection, I implemented sophisticated filtering algorithms to robustly detect calibration targets in both camera and LiDAR data.

Camera Data Processing:
  • Applied RANSAC (Random Sample Consensus) to robustly detect checkerboard patterns
  • Implemented sub-pixel corner detection for maximum precision
  • Handled challenging lighting conditions and shadows
LiDAR Data Processing:
  • Used DBSCAN clustering to identify checkerboard regions in 3D point clouds
  • Developed plane-fitting algorithms to extract checkerboard surfaces
  • Applied statistical outlier removal for clean target detection
3Mathematical Foundation: Transformation Estimation

The core mathematical challenge was computing precise 6-DOF transformations between sensor pairs. I developed a multi-stage approach combining classical computer vision techniques with machine learning optimization.

Initial Transformation Estimation:
  • Applied Singular Value Decomposition (SVD) for initial pose estimation
  • Leveraged principles of 3D view geometry to establish correspondence
  • Used linear algebra techniques to solve for rotation and translation matrices
Machine Learning-Based Fine-Tuning:
  • Implemented scikit-learn-based optimization pipeline
  • Minimized Euclidean distance between corresponding checkerboard centers
  • Created custom loss function weighting positional and angular errors
  • Used iterative refinement for sub-centimeter accuracy
4Advanced Geometric Analysis for Rotational Calibration

The most challenging aspect was determining precise rotational alignment. I developed a novel approach using surface normal analysis and iterative optimization.

Surface Normal Analysis:
  • Computed plane equations from filtered LiDAR point clouds
  • Calculated surface normals for each detected checkerboard plane
  • Applied cosine distance calculations between corresponding surface normals
Iterative Closest Point (ICP) Refinement:
  • Implemented ICP algorithm for LiDAR-to-LiDAR alignment
  • Used point-to-plane distance minimization for robust convergence
  • Applied multi-resolution approach for faster optimization
5Real-World Validation and Integration

To prove the system's effectiveness, I integrated the calibrated sensor suite into a complete autonomous vehicle perception stack, demonstrating practical applications.

Lane Detection Pipeline:
  • Developed real-time lane detection using C++ and TorchScript optimization
  • Achieved 12+ FPS performance on NVIDIA Jetson Xavier
  • Projected detected lanes onto HD maps for path planning
  • Demonstrated seamless sensor fusion for navigation tasks
Performance Metrics
  • Latency: <50ms end-to-end
  • Throughput: 12+ FPS
  • Accuracy: Sub-centimeter precision
  • Robustness: All weather conditions

Results and Impact

<4°

Rotational Error

<10cm

Translational Error

12+ FPS

Real-time Performance

System Applications:

This calibration system became the foundation for the entire autonomous vehicle perception stack, enabling:

  • Accurate object detection and tracking across sensor modalities
  • Precise localization and mapping for navigation
  • Safe navigation in complex environments
  • Reliable sensor fusion for decision-making

The methodology I developed has applications beyond autonomous vehicles, including robotics, industrial automation, and augmented reality systems where precise multi-sensor calibration is critical.

Technical Stack & Tools

Programming Languages
  • Python (Data processing, ML)
  • C++ (Performance optimization)
  • CUDA (GPU acceleration)
Frameworks & Libraries
  • ROS (Robot Operating System)
  • OpenCV (Computer Vision)
  • NumPy, SciPy (Numerical)
  • scikit-learn (ML optimization)
Algorithms & Methods
  • RANSAC (Robust estimation)
  • DBSCAN (Clustering)
  • ICP (Point cloud alignment)
  • SVD (Matrix decomposition)
Hardware & Tools
  • FLIR Pointgrey Cameras
  • Velodyne VLP-16 LiDAR
  • NVIDIA Jetson Xavier
  • RViz (Visualization)

Future Applications & Extensions

This work demonstrates my ability to tackle complex multi-disciplinary problems, combining theoretical knowledge with practical implementation to deliver robust, production-ready solutions. The calibration framework can be extended to:

  • Advanced Driver Assistance Systems (ADAS) for commercial vehicles
  • Industrial robotics for precision manufacturing
  • Augmented reality applications requiring precise spatial alignment
  • Surveillance systems with multiple camera and sensor arrays
  • Drone applications for aerial mapping and inspection