Applied Industrial AI for Cranes

Sensor Data

AutoCrane uses both visual data and LiDAR data. The incoming visual sensor data is processed (pre-processed) in parallel using a highly distributed system architecture. A separate process—known as a “sensor node”—is dedicated to each imaging sensor (currently a camera and 3D LiDAR).

A sensor node processes incoming data at high bandwidth and then transmits only the relevant processing results to the core system’s synchronous control loop at a significantly lower bandwidth. The sensor processing is designed to be completed entirely within a single control cycle.

Processing of visual data

The camera images are analyzed at a rate of 20 Hz by a multi-stage image processing chain. This chain is specifically tailored to the respective perspective and task in order to precisely extract the necessary information. While traditional machine vision algorithms are also used in the later stages of this chain, a neural network always constitutes the first step. In this way, we achieve a level of precision and resilience to lighting conditions, weather conditions, and other environmental changes that would be impossible to achieve in practice using traditional methods alone.

Processing of LiDAR Data

LiDAR data is used to create elevation maps and to precisely determine the position and shape of objects—especially when visual information (e.g., from cameras) is insufficient. The elevation maps are automatically generated from the incoming measurement data (the 3D point cloud from a single measurement) and continuously updated.

As an integral part of the global model, this data is used for collision avoidance, path planning, and the precise determination of drop-off and pickup points in the storage area. Known objects are reliably recognized in the point clouds, allowing their 6D pose (their exact position in space and orientation along all three spatial axes) to be determined.

This type of registration enables highly accurate localization and the determination of relative distances between multiple objects—for example, between the grapple and the load. AutoCraneI uses this data as an independent, secondary measurement system during truck unloading and as the primary source of information during train unloading.

In this process, the crane’s grapple and the train itself are registered in the point cloud using a variant of the “Iterative Closest Point” (ICP) algorithm. The 3D models required for this can either be created by scanning the objects beforehand with LIDAR sensors or extracted fully automatically from existing CAD models.

The Neural Networks We Use

The proprietary neural networks we use are based on the “All-Convolutional-Net” and “U-Net” architectures developed in Freiburg, Germany, which have proven themselves in a wide range of applications worldwide. The use of pre-trained standard networks (“off-the-shelf”) was ruled out due to the specific application area and highly specialized requirements.

Hohe Geschwindigkeit

In addition to handling unusual perspectives and rotating objects, we place particular emphasis on extremely low inference times (typically 5–15 ms) and remarkable data efficiency. Our networks and training algorithms typically require only about 20,000 images to reach full performance—compared to the 14 million images in the ImageNet dataset.