A Computer Vision System for Terrain Recognition and Object Detection Tasks in Mining and Construction Environments

Recent studies towards dragline excavation efficiency have focused on incrementally achieving automation of the entire excavation cycle. Initial efforts resulted in the development of an automated dragline swing system, which optimizes the swing phase time. However, the system still requires human operation for collision avoidance. For full dragline autonomy, a machine vision system is needed for collision prevention and big rock handling during the ‘swinging’ and ‘digging’ phases of the excavation operation. Previous attempts in this area focused on collision avoidance vision models which estimated the location of the bucket in space in real-time. However, these previous models use image segmentation methods that are neither scalable nor multi-purpose. In this study, a scalable and multi-purpose vision model has been developed for draglines using Convolutional Neural Networks. This vision system averages 82.6% classification accuracy and 91% detection in collision avoidance. It also achieves an 87.32% detection rate in bucket pose estimation tasks. In addition, it averages 80.9% precision and 91.3% recall performance across terrain recognition and oversized rock detection tasks. With minimal modification, the proposed vision system can be adjusted for other automated excavators. Keywords: Surface mining, dragline bucket, deep learning, machine learning, oversized rock, convolutional neural network, machine vision, object detection, earthmoving equipment. INTRODUCTION Complete excavator automation is widely regarded as the next phase in efforts toward improving earthmoving efficiency. Earthmoving often involves complex, forceful interactions between an excavator and the ground. The nature of these interactions depends largely on the type, properties and physical characteristics of the earth material. This process is further complicated by the random occurrence of tree roots, boulders and other such obstructions. Therefore, the ability of a controller to detect changes in the operating conditions, adjust the digging strategy and respond in real-time is of utmost importance. Early studies into autonomous excavation identified some key performance criteria which included the following [1]: • The autonomous excavator must be able to work in any type of earth material. • Its excavation accuracy must be within 50mm. • It should be able to handle different surface and underground obstacles autonomously. • It should be able to operate at the speed of the average operator in any condition. • Its operation should be capable of safe integration with other site systems.