Sumitomo Heavy Industries (SHI) and tech giant NEC are developing a system that automatically extracts recordings of near-miss incidents and generates corresponding reports based on the camera footage and sensor data captured from hydraulic excavators.
The system will use an extraction AI model, trained on real-world hydraulic excavator data accumulated by SHI, to first identify and extract risk scenes from video footage.
These risk scenes, together with operational data from the excavators, will then be analysed using NEC’s proprietary technology, which combines video recognition with generative AI, and stored as multimodal data that incorporates temporal and spatial information.
Based on this data, along with SHI’s knowledge of construction site machinery operations and human workflows, the system will cross-reference hazardous and prohibited behaviour data. These are defined by accidents, construction equipment failures and operations requiring particular attention, as well as company-specific data.
The system will automatically identify the risk scenes that should be reported and generate high-quality near-miss reports that summarise the circumstances surrounding each incident.
SHI and NEC plan to have the system commercially available in the 2027/28 financial year. In future, they plan to broaden the system’s application beyond worker-plant interfaces to include unsafe conditions that may not be readily recognised by workers, as well as considerations for site-specific operational rules.
