Autonomous maritime collision avoidance: Field verification of autonomous surface vehicle behavior in challenging scenarios

We present results from sea trials for an autonomous surface vehicle (ASV) equipped with a collision avoidance system based on model predictive control (MPC). The sea trials were performed in the North Sea as part of an ASV Challenge posed by Deltares through a Dutch initiative involving different authorities, including the Ministry of Infrastructure and Water Management, the Netherlands Coastguard, and the Royal Netherlands Navy. To allow an ASV to operate in a maritime environment governed by the International Regulations for Preventing Collisions at Sea (COLREGs), the ASV must be capable of complying with COLREGs. Therefore, the sea trials focused on verifying COLREGs‐compliant behavior of the ASV in different challenging scenarios using automatic identification system (AIS) data from other vessels. The scenarios cover situations where some obstacle vessels obey COLREGs and emergency situations where some obstacles make decisions that increase the risk of collision. The MPC‐based collision avoidance method evaluates a combined predicted collision and COLREGs‐compliance risk associated with each obstacle and chooses the ‘best’ way out of dangerous situations. The results from the verification exercise in the North Sea show that the MPC approach is capable of finding safe solutions in challenging situations, and in most cases demonstrates behaviors that are close to the expectations of an experienced mariner. According to Deltares’ report, the sea trials have shown in practice that the technical maturity of autonomous vessels is already more than expected.

In the quest for an answer to the question above, authorities in the Netherlands, together with Deltares (an independent institute for applied research in the field of water and subsurface), invited selected companies to undergo a verification exercise in the Dutch North Sea to demonstrate and validate the capabilities of autonomous surface vehicles (ASVs). The results of the verification exercise are intended to be used as a supporting material for the Netherlands Rijkswaterstaat's initiative to replace manned vessels with unmanned vessels for monitoring water quality in the Dutch North Sea. The goal of the Dutch initiative is to achieve an efficient, safe, and sustainable water monitoring program through the use of autonomous measurement platforms (Verheul, 2017(Verheul, , 2018. The success of the verification exercise will therefore speed up the acceptance and implementation of the desired autonomous water monitoring program. Maritime Robotics AS was among the companies invited to the Netherlands for the ASV verification exercise. Telemetron, which is Maritime Robotics' research and development vessel was therefore prepared for a test week in the North Sea, from November 20 to 24, 2017. In close collaboration with the Norwegian University of Science and Technology (NTNU), Maritime Robotics equipped Telemetron with the model predictive control (MPC)-based collision avoidance strategy described in Hagen, Kufoalor, Brekke, and Johansen (2018) and Johansen, Perez, and Cristofaro (2016). Since its first implementation and deployment on the Telemetron vessel in 2016, the MPC strategy has been tested and refined through experimental field work consisting of several campaigns in the Trondheimsfjord in Norway. Moreover, extensive field tests were carried out before the autonomous vehicle, Telemetron, was transported to the Netherlands for the verification exercise.
Since field reports available on testing COLREGs-compliance of autonomous collision avoidance methods are mainly based on the designer's/researcher's own planned experiments (see, e.g., Benjamin, Leonard, Curcio, & Newman, 2006;Hagen et al., 2018;Kufoalor, Wilthl, Hagen, Brekke, & Johansen, 2019;Kuwata, Wolf, Zarzhitsky, & Huntsberger, 2014;Schuster, Blaich, & Reuter, 2014;Svec et al., 2013) the Dutch verification exercise was a unique opportunity to validate the performance of collision avoidance methods in real maritime traffic, under the direction of experienced independent authorities. Bloot Nautical Consultancy was tasked by the Netherlands Directorate-General for Maritime Affairs to design a test plan for the verification exercise, whereas the Royal Netherlands Navy was responsible for the actual test validation process during the test week.
The maritime domain is characterized by several factors, including a large variety of obstacles, uncertain obstacle motion, complex interactions between vessels, and varying sea states.
However, most of the existing field reports focus on passive obstacle behaviors in controlled environments and ideal weather conditions. The field trials reported in Benjamin et al. (2006), Schuster et al. (2014), and Svec et al. (2013) cover a few basic COLREGs scenarios, involving low-speed (<3 m/s) and close-range encounters with a single-obstacle vessel. Some complex single and multidynamic obstacle scenarios are reported in Hagen et al. (2018), Kufoalor et al. (2019), and Kuwata et al. (2014). Apart from the reports of Hagen et al. (2018) and Kufoalor et al. (2019), field testing in varying and challenging weather conditions does not seem to be part of the validation process of autonomous maritime collision avoidance methods.
As revealed by the first DARPA Grand Challenge (Buehler, Iagnemma, & Singh, 2007) and several competitions in the maritime domain (e.g., The Microtransat Challenge;Microtransat, 2019), verification exercises outside the control of robotics researchers usually demonstrate that algorithms and robot systems that function perfectly in simulations or controlled experiments are not always effective in the real world. Therefore, an important motivation for participating in the Dutch verification exercise was to uncover important aspects of maritime autonomous collision avoidance that should direct further work and future research efforts.
This paper presents and discusses the field results achieved using a collision avoidance method based on MPC. The discussions focus on important factors that must be considered to achieve COLREGscompliance in challenging dynamic scenarios. The field test approach used during the verification exercise deviates from the typical approach used in research experiments, where test scenarios are usually limited to fixed preplanned behaviors of obstacle vessels. By adapting scenarios on the fly, challenging situations that developed due to an unexpected change in obstacle behavior were explored in the sea trials. This test approach, which considers spontaneous/ unrehearsed changes in obstacle behavior, may be adopted by researchers for field testing and benchmarking of maritime collision avoidance methods.
The remainder of this paper is structured as follows. We discuss relevant aspects of COLREGs in Section 2, with focus on the test plan used during the sea trials. Section 3 describes the ASV and the system architecture used for autonomy and remote control. Section 4 provides a description of the collision avoidance strategy before the results from the verification exercise are presented in Section 5. We discuss our observations and important lessons learned in Section 6, and we provide concluding remarks in Section 7.

| REQUIRED VESSEL BEHAVIOR ACCORDING TO COLREGS
An important goal of the sea trials was to test scenarios that verify the capability of an ASV to safely navigate autonomously in the North Sea (Verheul, 2017). Since no special rules and regulations existed for autonomous vehicles at the time of the tests, the main task of the ASV was to demonstrate compliance to the existing "rules of the road", COLREGs (Cockcroft & Lameijer, 2012;IMO, 1972), which is applicable to all marine vessels in the North Sea. Moreover, the ASV's behavior should meet the expectations of experienced mariners, in this case two Commander Lieutenants-at-sea (LTZ 2) from the Royal Netherlands Navy. Note that COLREGs was written for the human operator, and it is not straightforward to apply some aspects to an autonomous vessel. However, through a careful design of scenarios, it is possible to perform tests that reveal the ASV's "sense of responsibility" (Rule 2), situational awareness capabilities (Rule 5), evaluation of collision risk (Rule 7), and collision avoidance action plan (Rules 6,8,(13)(14)(15)(16)(17)(18)(19). The rules from COLREGs considered in the verification exercise are discussed in this section.

| Responsibility
Rule 2 of COLREGs holds all marine vessels responsible for their actions in both ordinary and special circumstances. All vessels are tasked to do everything possible to avoid collision, and if necessary, depart from the rules to avoid immediate danger. This rule requires a "sense of responsibility" that should also govern the collision avoidance strategy of the autonomous vessel. The ASV's sense of responsibility can be demonstrated/observed through its choice of actions, which should preferably be proactive.

| Situational awareness
Both Rules 2 and 5 emphasize the importance of understanding different situations in the maritime environment. Rule 2 states that "due regard shall be had to all dangers of navigation and collision and to any special circumstances, including the limitations of the vessels involved," and Rule 5 states that "every vessel shall at all times maintain proper lookout by sight and hearing as well as all available means appropriate in the prevailing circumstances and conditions so as to make appraisal of the situation and of the risk of collision." For an autonomous vehicle, these rules demand an appropriate system for accurate detection, identification, classification, and prediction of the effect of different factors in a complex dynamic maritime environment.

| Evaluation of collision risk
On the basis of the vessel's knowledge of a particular situation, an appraisal of the situation must be made and the risk of collision must be assessed as mentioned in Rule 5. This requirement is reemphasized in Rule 7, which states that "every vessel shall use all available means appropriate to the prevailing circumstances and conditions to determine if risk of collision exists." Further specifications in Rule 7 highlight the challenging aspects of risk assessment, where emphasis is put on the appropriate assessment of scanty information. For an automatic decision process to be possible, we need a useful way of quantifying risk based on possibly uncertain information.

| Collision avoidance actions
Rule 8 specifies the general behavior strategy every vessel should have and the required actions in dangerous situations. First of all, "any action taken to avoid collision shall be taken in accordance of the Rules." Recall that the rules cover both proactive and reactive actions.
Moreover, Rule 8 focuses on the properties of proactive actions, which include early and clear (i.e., large enough) alteration of course or speed intended to control a situation.
Controlling a situation at sea involves having due regard to the observance of good seamanship (see Rule 8(a)). In other words, the actions made should not make the situation worse for any other vessel in the vicinity. This clearly demands the complex task of understanding a situation from the perspective of both the own vessel and other vessels. Nevertheless, making early actions according to the rules will provide other vessels ample time to also choose actions aimed at reducing the risk of collision.
Required actions in different scenarios are specified in Rules 9-19 of COLREGs. Some of the actions verified during the sea trials are applicable to power-driven vessels at sea (i.e., IMO, 1972, Rules 13-19). An illustration of some basic scenarios can be seen in Figure 1.

| Complex collision avoidance scenarios
In a dangerous encounter between two vessels, the give-way vessel is required to keep out of the way of the other vessel by altering its course and/or speed to pass the other vessel at a safe distance (see IMO, 1972, Rule 8(f.i) and 16). The stand-on vessel is required to keep its course and speed (cf., Rule 17(a.i)). However, if the give-way vessel is not taking appropriate action in compliance with the rules, the stand-on vessel may take action to avoid collision (see Rule 17(a.ii)). The stand-on vessel is also obliged to take action when the situation is such that collision cannot be avoided by the action of the give-way vessel alone (cf., Rules 8(f.ii) and 17(b)).
The above statements from COLREGs illustrate that some scenarios are straightforward to interpret (cf., Figure 1) whereas others are complex. Complex scenarios can easily arise involving one or more obstacle vessels. For a single obstacle, a complex situation may arise at the boundary between two basic scenarios, especially when it becomes unclear which vessel is the give-way or stand-on vessel. To avoid such challenging scenarios, COLREGs requires a vessel that is the give-way vessel in a particular scenario to refrain from actions that result in switching between scenarios. A specific case is found in Rule 13 for overtaking, which states that "any subsequent alteration of the bearing between the two vessels shall not make the overtaking vessel a crossing vessel within the meaning of these Rules or relieve her of the duty of keeping clear of the overtaken vessel until she is finally past and clear." In general, the situation becomes complex when the behavior of the vessels involved becomes unpredictable. Unpredictable behaviors are even more challenging in multi-obstacle scenarios where some vessels obey the rules and others do not. We will present (in Section 5) both basic and complex scenarios from the sea trials that verify COLREGs-compliance in both single-and multi-obstacle encounters.

| Autonomous vessel
The ASV is Maritime Robotics' research and development vessel called Telemetron (shown in Figure 2a). Telemetron is a Polar Circle 845 Sport vessel, which is a stable and highly maneuverable Rigid Buoyancy Boat (RBB). It is type-approved as a class C vessel, and hence has an operational limitation of • wind speed up to 13.8 m/s, • wave height up to 2 m.
Some relevant specifications are shown in Table 1.
The ASV is equipped with several hardware and software components that make both remote control and autonomous navigation possible. The imaging sensors used for situational awareness and remote operation of the ASV during the sea trials can be seen in Figure 2b. Details of the sensor system and control architecture are presented next.
3.2 | ASV system architecture We used the automatic identification system (AIS) for obstacle motion sensing and tracking for autonomous collision avoidance decision making. Although obstacle tracking using AIS has reliability issues and depends on the accuracy of the obstacle vessel's global positioning system (GPS) and navigation system, it is sufficient for the sea trials because our main focus is on the behavior of the ASV in different challenging COLREGs scenarios.
Refer to Kufoalor et al. (2019) for later extensions to radar-based tracking.

| Obstacle vessels
The ASV's control system architecture, showing the main components of the OBS, remote link to the VCS, and the flow of information (cf., Figure 3). In this setup, only AIS obstacles are considered in the autonomous decision-making process. Information about radar, camera, and mapped obstacles is made available at the VCS to aid remote control decisions. AIS, automatic identification system; ASV, autonomous surface vehicle; OBS, on-board system; VCS, vehicle control station KUFOALOR ET AL.

| 391
We provide a brief description of the cost function components c f g , , , , Essentially, formulation (1) states that the optimal control behavior * k in a dangerous situation is the control behavior k that results in the minimum worst-case hazard. The cost function k expresses the hazard associated with selecting a control behavior k defined by course ( k m χ ) and speed (u k m ) modifications that are applied to corresponding desired reference values, u , d d χ , for the course ( χ ) and speed (u), respectively. We use the following set of alternative control behaviors, which we assume to be fixed on the prediction horizon: • course offset in degrees: 0, 15, 30, 45, 60, 75, 90 ; • speed factor: We use a minimum of ∘ 15 course offset to ensure that a change in course by the ASV is clear and readily apparent to other vessels observing visually or by radar (cf., Rule 8b of COLREGs). The use of speed factors, instead of fixed speed offsets, ensures that changes in speed are larger for high vessel speeds than that for low speeds. We also avoid increasing speed beyond the desired reference speed in dangerous situations.
The function c i t , in (1) denotes the cost of colliding with obstacle i at time t, considering a collision risk that depends on the time and distance to the CPA and scales with the relative velocity ( − v v t k i t k , ) of the ASV and obstacle i: The allowed CPA (i.e., d i min ) is defined by a safety distance parameter (d safe ) and the obstacle's length (L i ). Specifically, is used to define the radius of a circular safety region, which encloses obstacle i.
In (2) (1) is a grounding cost that penalizes control behaviors that will result in collision with land or defined no-go zones.
The cost for each control behavior k at time ∈ ( ) t t 0 is calculated based on the predicted state of the ASV and each obstacle i, obtained from the simulation of their trajectories. The fast dynamics of the ASV allows us to use the following simple kinematic model for predicting the ASV's future motion: χ denotes the position and course in the earth-fixed x y represents the velocities in surge, sway, and yaw specified in the body-fixed frame, and ( ) R χ is a rotation matrix from body-fixed to earth-fixed frame. The trajectory prediction is achieved

| Inherent properties and robustness
The MPC collision avoidance method described above prioritizes straight-line motion, which is considered as predictable behavior in a maritime environment. The strategy is to seek the least conservative solution according to the given constraints. This is achieved by prioritizing solutions that result in a tangential motion with respect to the boundary of the projected combined safety region associated with the obstacle vessels.
Due to the implementation of a COLREGs transitional cost (⋅) it is straightforward to prioritize COLREGs-compliant maneuvers in long-range encounters. Moreover, using a collision cost (⋅) c i t , that scales with the collision time, range, and relative velocity (cf., (2)), ensures that the MPC strategy will choose an evasive maneuver if collision becomes imminent.
Another important property of the MPC strategy is its inherent robustness to noise or uncertainty. All potentially uncertain variables that affect the collision avoidance decisions are evaluated in the cost function k over a long prediction horizon T . In combination with an appropriate choice of sampling time T s and a scenario grid of alternative control behaviors, the cost function provides a filtering effect that ensures that changes in each variable must be significant enough to produce a change in the decisions. Moreover, the collision cost (⋅) (2) prioritizes avoiding collision hazards that are close in time over those that are more distant and usually more uncertain .

| Practical aspects: tuning and operational modes
The MPC-based collision avoidance system is a simplification of the more general method presented in Johansen et al. (2016). The method depends on several parameters that must be selected carefully to achieve the desired ASV behavior in different scenarios. For instance, we have specified a larger set of alternative modifications for the ASV's course k m χ than that for speed u k m (see Section 4.1), and we tuned the cost function in (1) to prioritize course changes over speed. This ensures that the actions of the ASV in a dangerous situation are clear and easy to observe by other vessels, especially those operated by humans (cf., COLREGs Rule 8).
By implementing a COLREGs transitional cost (Hagen et al., 2018), we avoid the implementation of a specific guidance law in the prediction model of the MPC. We assume that the desired reference values from the guidance system remain constant over the prediction horizon, and we depend on the transitional cost to avoid unnecessary KUFOALOR ET AL.

| 393
switching of control behavior due to discrepancies between the actual guidance dynamics and the constant model. An alternative approach is to implement a model-switching strategy for each guidance strategy installed on the ASV, for example, line-of-sight (LOS) path following, pure pursuit (waypoint target), or course-/ heading-hold control. • open sea mode, • near coast mode, • narrow channel mode, • harbor mode, • mission-specific mode.
The appropriate mode can be selected by either an operator or automatically based on the ASV's environment/location and operational status. At sea, it is natural to use larger parameter values for the various distances and prediction horizon, compared with near coast or harbor modes. Some channels may be narrow or governed by traffic separation schemes, requiring appropriate adaptations of the ASV's behavior, while a specific ASV mission/operation may require a particular collision avoidance behavior. An example is the case where the ASV is supposed to follow a preplanned pattern for survey purposes, similar to the expectations of the proposed autonomous water monitoring program for the Dutch North Sea (see Section 1). In this case, it may be desirable to reduce the allowed offset limit from the planned path and prioritize the use of speed modifications for collision avoidance. Although the collision avoidance system was tuned for open sea and near coast modes, only the near coast mode was verified during the sea trials presented in this paper.

| Location and environmental conditions
To demonstrate that the ASV is capable of navigating autonomously in the Dutch waters of the North Sea, a test location (400 km 2 ) to the west of Texel, in the Netherlands, was assigned by the Department of Maritime Affairs, Coastguard, and the Traffic Control Center in Den Helder. This test location is a relatively low-traffic area, where large sea-room is available for collision avoidance maneuvers, and therefore allowing the "Open sea mode" parameterization of the collision avoidance system to be used. Considering the size and maneuvering capabilities of the vessels involved in the sea trials, a safety distance (d safe ) of 0.2 nautical mile (NM) and a COLREGs-compliant decision initiation distance (d init ) of 4 NM was required.
However, due to extreme weather conditions (with wind speeds of 10-28 m/s) in the assigned test area, the test location was moved from the west side of Texel to the east side in the Wadden Sea. Extracts from detailed weather forecast reports from Day 2 of the test week can be seen in Figure 5. Note that the weather conditions in the Wadden Sea were still rough enough to challenge the certified limits of the ASV Telemetron, which can operate in sea states of up to 2 m wave heights (i.e., the light green zone in Figure 5a) and 13.8 m/s wind speeds (i.e., max 6 on the beaufort scale in Figure 5b). The new test area is also closer to shore, with more traffic and limited sea-room for collision avoidance maneuvers. Therefore, we used a set of parameters for the "Near coast mode" with a reduced safety distance of 0.1 NM and a COLREGs-compliant decision distance of 1 NM. The same set of parameters was used for all the test scenarios.

| Test scenarios
Different challenging scenarios that verify compliance with the applicable rules of COLREGs discussed in Section 2 are presented in this section. The scenarios were set up using remote commands supervised by two LTZ 2 of the Royal Netherlands Navy at the VCS on the Coastguard vessel Zirfaea (see Figure 3). The Navy's assessment of each test scenario can be found in Verheul (2017, Appendix G), and we use the same title given to some scenarios during the verification exercise. Due to the weather conditions and long distances considered during the trials, it was not possible to rely on lookout from Zirfaea for remote control and monitoring of most scenarios. Reliability of the VCS was therefore a crucial factor in the verification process.
A screenshot from the VCS computer is shown in Figure 6, where relevant information about the ASV's motion, obstacles, and collision danger are highlighted. The vessels officially involved in the sea trials, with boxed labels in Figure 6, were given planned routes at the beginning of each scenario, whereas the other vessels seen at the VCS entered/exited scenarios at will, providing different interesting and spontaneous collision avoidance situations for the ASV. As an introduction to the dynamic nature of the test scenarios and the performance of the ASV, we will start with a brief description of the first test, which is captured in Figure 6.
The multi-obstacle scenario in Figure 6 started with the ASV traveling at 10 kn on the brown straight-line path parallel to the coast line. The ASV had to first avoid colliding with the FRISC, which was crossing from the ASV's starboard along the blue highlighted path. The scenario developed quickly into a crossing and head-on situation with another vessel, ELSE JEANNETTE, which entered the test area unannounced. The COLREGs-compliant behavior of the ASV can be seen by its trail shown in green. The ASV prioritized COLREGs-compliance, especially predictability, instead of attempting to steer towards its original path. At the current position of the vessels shown in the VCS, the FRISC accelerated to 40.7 kn (see AIS object information in Figure 6) creating a dangerous situation predicted ahead of the ASV and involving ELSE JEANNETTE, which was close by. Nevertheless, the highly uncertain behavior of the FRISC did not lead to unpredictable course changes by the ASV.
We will take a closer look at the vehicle speed, course, position trajectories, and control variables in the scenarios presented next.

| Crossing scenarios
In Figure 6, we saw a crossing scenario where the ASV was the giveway vessel and therefore took full responsibility in avoiding collision.  The movement of TH4 ELIZABETH can be seen in Figure 8a, from point p 2 . Although TH4 ELIZABETH approached in a crossing scenario, and may therefore expect a stand-on behavior by both the ASV and FRISC, the risk of collision with FRISC was much higher for the ASV. Moreover, choosing a clear starboard maneuver, instead of a course change towards port or speed reduction, is reasonable in this situation. Note that the ASV returned to its original path after crossing ahead of TH4 ELIZABETH, with a distance of more than 1,000 m between the two vessels.

| Combined overtaking and crossing scenarios
The results in Figure 9 were obtained from a complex scenario where the ASV had to overtake ZIRFAEA while keeping well clear of FRISC, which was crossing the paths of both ZIRFAEA and the ASV. As the crossing and overtaking situation evolved between points p 1 and p 2 (see Figure 9a), the ASV had to explore the space between ZIRFAEA and FRISC by keeping a safe distance (at least 0.1 NM) and acting predictably according to COLREGs. Note that the ASV has passed between ZIRFAEA and FRISC at p 2 . After point p 2 , the ASV was steering gradually back to its original path when FRISC decided to approach ZIRFAEA. The approach speed and motion of FRISC (see Figure 9d) was associated with a high risk of collision in the ASV's decision method, and the ASV had to steer away briefly before returning to its original path when FRISC slowed down.

| COLREGs-decision boundary test scenarios
We will now examine the properties of the collision avoidance strategy of the ASV when the boundaries defining different COLREGs scenarios are challenged. More specifically, the following scenarios describe different situations where the obstacle vessel violates COLREGs by switching between overtaking and crossing scenarios (see [Rule 13]; IMO, 1972 and Section 2.2). During the verification exercise, we referred to the scenarios discussed next as "Navy pursuit evasion" and "Navy restriction" scenarios.

| Navy pursuit evasion scenario
In the test shown in Figure 10, FRISC approached the ASV from behind, targeting a bearing of ∘ 112.5 , which defines the boundary between the overtaking and crossing regions according to COLREGs. The ASV's speed was 8 kn (~4 m/s), whereas FRISC was traveling at 16 kn (~8 m/s, cf., speed curves in Figure 10b,c). At point p 1 in Figure 10a, the vessels are 600 m apart, and FRISC is in the ASV's overtaking region with a predicted trajectory passing in front of the ASV. Since the predicted CPA violates the ASV's desired safety distance, it steers towards port to create enough sea-room for FRISC to pass. Note that the ASV stands on for a while (for predictability according to COLREGs) before maneuvering at point p 1 . After running almost parallel for about 2 min and reducing the separation distance to 500 m, FRISC made a rapid speed reduction to about 6 kn (~3 m/s seen in Figure 10c) and steered towards port. The action of FRISC changed the situation from an overtaking scenario to a crossing scenario.
Since FRISC now appears to be crossing behind the ASV at a lower speed, the ASV steered towards its original path. However, FRISC returned to the overtaking region and started closing in on the ASV at 13 kn (~6.5 m/s). At point p 2 in Figure 10a, the distance between the two vessels is 250 m, and very close to the ASV's safety region. It can be seen in Figure 10 that the ASV's efforts towards the end of the scenario became purely evasive, but it still maintained a predictable behavior.

| Navy restriction scenario
In contrast to the scenario described above, the next scenario shown in Figure 11 started with FRISC traveling ahead at 12 kn (~6 m/s seen in Figure 11c), and the ASV was crossing behind FRISC at a lower speed of 8 kn (~4 m/s in Figure 11b). FRISC kept its course almost constant throughout the scenario. However, FRISC was ordered to reduce its speed to 6 kn (~3 m/s) around point p 1 (see Figure 11c) when the ASV was just 300 m behind. Considering the ∘ 112.5 bearing boundary for overtaking and crossing from FRISC's perspective, the crossing scenario changed into an overtaking scenario. Although FRISC violated COLREGs at that moment, the ASV was obliged by COLREGs Rule 2 to take necessary actions to avoid collision. The ASV therefore made a clear and predictable starboard maneuver as seen between points p 1 and p 2 in Figure 11a,b. After point p 2 , FRISC speed up again to 16 kn, and the ASV could steer towards its original path.

| Obstacle intention uncertainty scenarios
A challenging factor that affects the behavior of the ASV is the uncertainty of the future motion of dynamic obstacles. This is in general due to the unknown, and in some cases unpredictable, intention of other vessels. Figures 12 and 13 present results from different trials that verify the ASV's behavior in situations where the future motion of FRISC is uncertain.
A complex and uncertain situation is shown in Figure 12 with FRISC crossing from port and ZIRFAEA crossing from starboard. The ASV is the give-way vessel to ZIRFAEA and the stand-on vessel to FRISC. ZIRFAEA is a stand-on vessel to the ASV and a give-way vessel to FRISC, which must give way to both vessels. This scenario shows how complex the COLREGs-decision process can become when considering more than one obstacle. Depending on which distance and speed each vessel considers to be safe, and how each vessel assesses the situation, the outcome will vary.
In Figure 12, we see the case where FRISC enters an ongoing crossing situation between the ASV and ZIRFAEA (cf., point p 1 in Figure 12a). Both ZIRFAEA and the ASV were moving with 6 kn (~3 m/s), whereas FRISC accelerated to almost 20 kn (~10 m/s) when detected at p 1 . The dangerous speed and course of FRISC makes its intention highly uncertain to the ASV. As a consequence, the ASV reacted by reducing speed and steering port. A desirable behavior is probably to continue steering starboard. However, the tuning of the MPC method allowed a purely evasive action to be selected by the ASV. Note that a straight-line prediction of the motion of FRISC suggests it will pass in front of the ASV and dangerously breach the required CPA distance, if FRISC does nothing about the situation. The reasoning behind the ASV's decision is to avert the dangerous situation by creating enough sea-room for FRISC to pass while still crossing behind ZIRFAEA.

| Navy versus Fisherman scenario
The last scenario shown in Figure 13 tests the case where unexpected actions of FRISC increase the risk of collision in a head-on situation.
During the test, we referred to this scenario as "Navy versus Fisherman". The reason is that the behavior of the obstacle vessel FRISC mimics the behavior of some fishing vessels in the North Sea.
The Navy's observation is that a fishing vessel may communicate its intention and later deviate from it. This leads to an unexpected situation that may be more dangerous than the case where no communication is made. The same situation occurs when the behavior of the obstacle vessel is clear enough to communicate an intention, and then the obstacle suddenly makes a significant change that contradicts the communicated intention.
The scenario in Figure

| Tracking of obstacles based on AIS data
The results presented in Section 5 were achieved based on AIS data received during the tests. The accuracy of the AIS measurements depends on the obstacle vessel's navigation sensors or positioning system, and the accuracy is generally unknown to the ASV. The variations in course and speed measurements seen in Figure 8d, 9d, and 12d for different obstacle vessels show that the accuracy of the measurements received is different for each obstacle. However, for the same weather conditions, the measurements received from larger vessels (e.g., ZIRFAEA in Figure 9d and TH4 ELIZABETH in Figure 8d) are less noisy compared with the measurements of the FRISC.
A crucial aspect of the AIS measurements is the update frequency. The AIS data received is typically updated at least once in 10 s, but the update rate may vary significantly depending on the

| Obstacle tracking fault tolerance
Apart from the possible inaccuracies of AIS measurements discussed in Section 6.2, the transmitted AIS data may be completely wrong.
The obstacle vessel may not have AIS or decide to switch off its AIS.
The AIS signal may be lost due to transmission faults or interference, and large transmission delays may render the AIS information useless for collision avoidance, especially in close-range encounters.
Further development and research progress in robustness and fault tolerance, using the ASV Telemetron as a case study, are KUFOALOR ET AL.

| Safety verification and assurance
Although the verification exercise did not focus on fault tolerance, the above observations and lessons learned from the sea trials motivate more research effort into safe, robust, and fault-tolerant autonomous collision avoidance. Moreover, test procedures that aim at providing safety verification and assurance must be able to verify the effects of uncertainty and faults on the behavior of autonomous vessels.
Due to the characteristics of the maritime domain and the associated large variety of dangerous situations that may arise, it is difficult to achieve safety assurance through field verification only.
To provide significant statistical evidence of the safety level of autonomous vessels, further research and development is needed in the area of automated verification, where a large variety and a significant number of scenarios can be generated and tested in a simulated environment. In Vartdal and Skjong (2018), DNV GL discusses the requirements and verification procedures needed to provide safety assurance of autonomous vessels, and they highlight the potentials of simulator-based verification. It is however recommended that simulator-based verification is complemented by dedicated field testing for validation purposes.
Note that the simulation approach requires adequate environmental models, vehicle models, and scenario generators that produce relevant test situations from operational specifications, COLREGs, and safety requirements. It is also necessary to develop useful verification metrics that adequately reflect the requirements of COLREGs and assess the general safety level of autonomous vessels.
Deriving such metrics in the context of COLREGs-compliance is not trivial, especially for complex scenarios (see Section 2.2) and for requirements that rely heavily on the practice of "good seamanship".
Nevertheless, research efforts, such as Woerner (2016), enable the development of new standards and procedures for automated verification of autonomous vessels.

| CONCLUSION
We have presented results from a verification exercise for ASVs in the Dutch North Sea. Both the ASV system architecture and the MPC-based collision avoidance method used were presented.
The discussions focused on COLREGs-compliance and the related challenges considering autonomous collision avoidance decision making. The test scenarios presented in this paper cover both typical situations at sea and complex or emergency situations that test the reasoning capabilities of the collision avoidance method.
In contrast to the typical test approach used in research experiments, the field tests in this paper were not carried out by constraining the scenarios to only fixed preplanned behaviors of obstacle vessels. Rather, real traffic scenarios were achieved by allowing regular traffic vessels to enter and exit planned scenarios unannounced. Some scenarios were adapted on the fly by supervisors from the Navy with the aim of exploring challenging situations that may develop due to an unexpected change in behavior. The test approach used by the Navy resulted in interesting scenarios that may be adopted by researchers for field testing and benchmarking of maritime collision avoidance methods.
The MPC-based collision avoidance method tested during the sea trials was able to find safe solutions in challenging situations, and in most cases the ASV demonstrated behaviors that are close to the expectations of the experienced mariners involved in the validation process. The field results are however not sufficient to provide general safety assurance. Different observations from the sea trials that motivate further work on robustness and fault tolerance are discussed in this paper, and we emphasize the need for further work on safety verification of autonomous vessels.
Finally, the achievements of the field verification exercise in the North Sea reveal that the technical maturity of autonomous vessels is approaching a level that will make their deployment desirable in the near future.