Fusing Event-based and RGB camera for Robust Object Detection in Adverse Conditions

Abhishek Tomy (1) , Anshul Paigwar (1) , Khushdeep Singh Mann (1) , Alessandro Renzaglia (1) , Christian Laugier (1)

1 CHROMA - Robots coopératifs et adaptés à la présence humaine en environnements dynamiques

The ability to detect objects, under image corruptions and different weather conditions is vital for deep learning models especially when applied to real-world applications such as autonomous driving. Traditional RGB-based detection fails under these conditions and it is thus important to design a sensor suite that is redundant to failures of the primary frame-based detection. Event-based cameras can complement frame-based cameras in low-light conditions and high dynamic range scenarios that an autonomous vehicle can encounter during navigation. Accordingly, we propose a redundant sensor fusion model of event-based and frame-based cameras that is robust to common image corruptions. The method utilizes a voxel grid representation for events as input and proposes a two-parallel feature extractor network for frames and events. Our sensor fusion approach is more robust to corruptions by over 30% compared to only frame-based detections and outperforms the only event-based detection. The model is trained and evaluated on the publicly released DSEC dataset.

Domaines: Informatique [cs]

Abhishek Tomy, Anshul Paigwar, Khushdeep Singh Mann, Alessandro Renzaglia, Christian Laugier. Fusing Event-based and RGB camera for Robust Object Detection in Adverse Conditions. ICRA 2022 - IEEE International Conference on Robotics and Automation, May 2022, Philadelphia, United States. ⟨hal-03591717⟩