Development of an intelligent framework for semantic road segmentation using depth and camera motion constraints: An approach for resource management and engineering decision making in transportation

Abstract

One of the major challenges in semantic segmentation of road videos is to effectively exploit the 3D structure of the scene and camera motion, while only 2D pixel labeling is available. In this paper, we introduce a semi-supervised learning framework for semantic segmentation that exploits geometric constraints imposed by depth and camera motion in the video. In this approach, for each 15-frame sequence, only the first frame is manually labeled pixel-wise, and for subsequent frames, pseudo-labels are generated by “geometrically mapping” the labeled frame mask to other frames. To estimate monocular depth and relative camera motion between frames, the Monodepth2 model is used, which performs self-supervised learning of depth and camera motion using new view reconstruction and photometric reprojection error. Based on the estimated depth and motion, each pixel from the reference frame mask is mapped to 3D space and then to the destination frame, and by comparing the mapped depth and the estimated depth in the destination frame, a validation mask is constructed to refine the pseudo-labels. These pseudo-labels are used for semi-supervised training of the same LR-ASPP segmentation network. Evaluation on the KITTI-STEP dataset shows that the semi-supervised model based on geometric mapping (SSL-Warp) achieves mIoU of 25.73% and pixel accuracy of 71.42%, providing a slight improvement in performance over the baseline frame-only model. Analysis of the results shows that the depth and camera motion constraints are very useful for static classes such as road and vegetation, but for objects with independent motion, they require combination with explicit motion information.