Outdoor scene understanding plays a key role for unmanned ground vehicles (UGVs) to navigate in complex urban environments. This paper presents a novel 3D scene-understanding framework for UGVs to handle uncertain and changing lighting conditions outdoors. A 2D bearing angle (BA) image is deployed to perform scene understanding so that the computational burden in the process of segmentation and classification of the 3D laser point cloud can be reduced. An improved super-pixel algorithm is used for fast 3D scene segmentation, and then the Gentle-Adaboost algorithm is utilized to perform super-pixel patch classification using the texture features of the Gray Level Co-occurrence Matrix. All false classification results in the uncertain super-pixel patches of BA images are transformed back to raw 3D laser points and a re-classification is conducted to refine the 3D scene understanding for UGVs. The results from a real laser dataset taken from a large-scale campus environment show the validity and robust performance of the proposed approach, in comparison with the results from Korea Advanced Institute of Science and Technology dataset.