One of the most important tasks when navigating a vessel is avoiding other vessels. A ship captain can rely on tools like the Automatic Identification System (AIS), radar, and communication with other vessels, but they ultimately use their vision to assess the situation. An autonomous vessel must replicate these capabilities. To safely navigate narrow and crowded inland waterways, autonomous ships must perceive other vessels and react in real time. My research has been focusing on achieving this by using a technology called Extended Object Tracking (EOT), which combines data from multiple sensors to help autonomous vessels “see” other vessels, not just as points, but in full detail, including their size, shape, and movement.
What is Extended Object Tracking?
Most traditional tracking systems treat obstacles as single points—this works in some scenarios but is inadequate when dealing with more complex objects, like ships at close range, especially in confined spaces like inland waterways. EOT goes beyond this by tracking the entire shape and size of an object. Instead of seeing another barge as just a point on a map, EOT builds a detailed picture of its outline, including its dimensions, shape, and how it moves. This level of detail is crucial for safely navigating in narrow waterways, where there’s limited room to maneuver around other vessels.
How Does EOT Work?
EOT relies on sensors, such as LiDAR and radar, to gather real-time data about the surrounding environment. These sensors scan nearby vessels, collecting multiple data points. The EOT algorithm then uses these measurements in conjunction with statistical models to generate estimates of not only an object’s location but also its size, shape, and speed. By integrating the sensor data with statistical models, the system refines its understanding of each object over time. Additionally, these models help assess the uncertainty of the estimates, providing the navigation system with crucial information about the reliability of the data.
Figure 1: An Extended Object Tracker takes the measurements (black circles) of a ship (shown in orange) to make an estimate (purple) and an uncertainty of said estimate (yellow shaded area)
What is Multiple Extended Object Tracking (MEOT)?
Multiple Extended Object Tracking (MEOT) involves tracking several extended objects simultaneously. While this may seem similar to tracking a single object, it introduces significant complexity. In a busy environment, such as an inland waterway, it becomes challenging to determine which measurements correspond to which object. Additionally, the number of vessels may be uncertain, as raw sensor data often includes clutter or irrelevant information.
To navigate this complexity, an MEOT algorithm uses the same statistical models employed in EOT to analyze measurements and determine the probability that each measurement originates from a specific object. When we receive data from a sensor scan, each measurement could potentially come from any of the tracked objects, an untracked new object, or from clutter.
To effectively track these vessels, the algorithm must assign each measurement to its corresponding source. Each assignment comes with a specific probability, allowing the algorithm to make informed decisions about which measurements likely belong to which object. With every measurement assigned, we get a data association hypothesis. For each sensor scan, there are many possible data association hypotheses. An MEOT algorithm handles these different data association hypotheses according to their respective probabilities.
Figure 2: A multiple extended object tracking algorithm can track multiple extended objects simultaneously by assigning the measurements to tracked objects. Demonstration of the algorithm is based on [1].
My Research
My research has focused on applying these concepts to the maritime domain, and specifically to inland waterways. In my first article, I presented a MEOT algorithm capable of tracking smaller vessels using LiDAR data [1]. I have since collaborated with a colleague at NTNU, who has incorporated data from AIS into multi-object tracking, to develop an MEOT algorithm that fuses data from both LiDAR and AIS to effectively track vessels [2].
To demonstrate the viability of this algorithm in a practical setting, I applied it to track real barges sailing in the Albert Canal in Belgium [3]. This real-world application highlights the potential of using MEOT to track vessels in an inland waterway.
Figure 3: Utilizing a multiple extended object tracker to track river barges using LiDAR and AIS sensor data from a shore-based sensor, as presented in [3]. The image is taken from the same place as the LiDAR data, the large black dot in the bottom of the figure.
In summary, EOT is a technology that is of vital importance to detect and track other vessels in an inland waterway, or in confined waters in general. To make autonomous ships a reality, technologies like EOT will be essential for ensuring safety and efficiency on busy inland waterways. While further work is needed to refine these models, I remain optimistic about the future of this technology and plan to continue pursuing this research direction to contribute to the advancement of autonomous navigation.
An article by Martin Baerveldt.
References
[1] M. Baerveldt, M. E. Lopez, and E. F. Brekke, “Extended target PMBM tracker with a Gaussian Process target model on LiDAR data,” presented at the 2023 26th International Conference on Information Fusion (Fusion), Jun. 2023, p. 8.
[2] A. G. Hem, M. Baerveldt, and E. Brekke, “PMBM Filtering With Fusion of Target-Provided and Exteroceptive Measurements: Applications to Maritime Point and Extended Object Tracking,” IEEE Access, vol. 11/2023, 2023, doi: 0.1109/ACCESS.2023.0322000.
[3] M. Baerveldt, J. Shuai, and E. F. Brekke, “Improved Fusion of AIS Data for Multiple Extended Object Tracking,” in 2024 27th International Conference on Information Fusion (FUSION), Jul. 2024, pp. 1–8. doi: 10.23919/FUSION59988.2024.10706459.