7.1. Range of Laser Scanner
We wanted to determine the maximum range that the laser scanner needs to cover to detect all the dangerous situations the bus is exposed to during normal operations. For this, we created a density plot of the location of all the warnings, based on the warnings from all good runs. The density plot for the PAT (Port Authority Pittsburgh) bus is shown in Figure 13, where the highest density is dark red and the lowest is dark blue.
Most of the warnings are located alongside the bus. On the right side are mostly pedestrians, who are moving toward the bus when the bus is coming to a stop. On the left side, most of the warnings are caused by passing vehicles. For the PAT bus there is a high concentration of warnings in the middle of the bus adjacent to the right side.
The latter is the location of the laser scanner, and these warnings were generated when the laser scanner was dirty. The warnings in the front right or front left area of the bus are generated when the bus is turning and an object is in the path of the bus.
The figure also shows two enveloping areas, which include 80% and 98% of all warnings, respectively. They can be described as rectangular boxes on the side of the bus extending 3 m (5 m) from the back and 3 m (5 m) to the side and a half-circle in front with a radius of 10 m (15 m). The system covers the area of a half-circle of 50 m radius for large objects (>1 m as viewed from the scanner), which is much larger than the area indicated by the enveloping limits. For pedestrian-sized objects, which are harder to detect and track, the coverage is approximately a half-circle of 20 m radius, which still includes the enveloping area. We can therefore be quite confident that we did not miss warnings because of a lack of coverage.
The enveloped areas give an indication of what the coverage of a commercial system should be. It is desirable for the sensor to have a range somewhat greater than the indicated area, because this enables the detection and tracking of objects before they enter the area.
7.1.2. SCWS Warning Algorithm
The sensors and modules described in the previous sections provide the dynamic quantities of the bus and the observed objects and additional information about the environment. These measurements are combined with preloaded information to analyze the threat level of the situation. In the warning algorithm, the system calculates the probability that a collision will occur within the next five seconds. If the probability of collision exceeds a certain threshold, an appropriate warning is displayed to the driver. In the warning algorithm for the SCWS, we have two warning levels: “alert” and “imminent warning.” An “alert” is displayed to the driver when the situation is somewhat dangerous, an “imminent warning” is given if the situation is dangerous enough to inform the driver in an intrusive way. A detailed description of the algorithm can be found in Mertz (2004). A short example is illustrated here.
In Figure 14, a bus turns right while an object crosses its path from right to left (World). The sensors measure the speed and turn rate of the bus and the location and velocity of the object. The algorithm calculates possible paths of the object with respect to the bus (Fixed bus). In this calculation, the paths are distributed according to the uncertainties of the measured dynamic quantities as well as according to models of driver and object behavior. These models are limits on speed, acceleration, turn-rate, etc. and were determined from previously recorded data. Next, the system determines for times up to 5 s into the future which fraction of these paths lead to a collision. In Figure 14, this is shown for the times 2, 3, and 5 s. This fraction is the probability of collision and is plotted versus time in Figure 15. This graph is divided into three areas, each a different level of threat severity. The area with the severest level that the probability of collision curve reaches determines the warning issued to the driver.
Figure 14. The trajectories of a bus and an object shown in the world coordinate frame (left) and the fixed bus frame (right). In the right figure, possible positions of the object are shown for the times 2, 3, and 5 s in the future. Light gray indicates that no collision has happened; dark gray indicates that a collision has happened.
Download figure to PowerPoint
Figure 15. Probability of collision plotted versus time. The three regions correspond to the warning levels aware, alert, and imminent.
Download figure to PowerPoint
The algorithm can also deal with environmental information. For example, if the object is a pedestrian and is on the sidewalk, there is an enhanced likelihood that the pedestrian will stay on the sidewalk. This is addressed by giving the paths leaving the sidewalk a lower weight.
Underbus warning. Another important alarm is the underbus warning. It is issued when a person falls and has the potential of sliding under the bus. We detect these situations by observing pedestrians who disappear while being close to the bus. The challenge in this algorithm is to distinguish people who disappear through falling and people who only seem to disappear, but in fact either merged with another object or are occluded by other objects.
Notification that a collision occurred. Sometimes the bus can collide with an object, especially a person, and the driver does not notice it. It is therefore important to notify the driver if a collision has occurred. A notification will be triggered if the probability of collision is 100% for times up to 0.5 s.
7.1.3. Missed Warnings
Alert and imminent warnings. The false negative alarm rate (missed warnings) for alerts or imminent warnings is difficult to determine because it is time-consuming to review large sets of data to find situations when warnings should have been given. Instead, we staged collisions to determine how many the system missed. We did not observe any missed warnings in these staged scenarios, which puts an upper limit of 16% on the ratio of missed warnings to correct warnings.
Failures that we observed in other situations that could lead to missed warnings are:
The system needs time to start up, so during this time no warnings can be issued.
The system reboots for some reason.
The laser scanner is retracted.
The laser scanner is dirty.
The bus is leaning and therefore the laser scanner does not point horizontally.
Some objects do not reflect the laser light sufficiently. For example, we observed a situation when a person was wearing a dark outfit on a rainy day and was not seen by the scanner. It could be that wet clothing specularly reflected the laser light or the dark clothing absorbed it.
Contact warnings. Initial evaluation showed that this algorithm is too restrictive, since objects colliding with the bus at small velocities do not trigger a contact warning. The system always gives an object an uncertainty in position and velocity, so the probability-of-collision calculation will not give a result of 100% unless the velocity of the object is sufficiently large and aimed at the bus.
Under the bus warnings. We used the data from these staged scenarios to test and calibrate the under-the-bus warning algorithm. We found that we needed to modify the tracking module (DATMO) to determine if an object went into occlusion or merged with another object. This information was passed to the warning algorithm so that it did not falsely think an object disappeared (and potentially was under the bus) when it in fact became occluded or merged with some other object. We also discovered that we should only consider objects that were detected for at least half a second to suppress alarms for spurious objects. Lastly, objects that are as far as 1.8 m from the bus when they disappear need to be considered. After these modifications and tuning of parameters, all 12 staged events gave correct under-the-bus warnings.
There were not enough staged events to determine a reliable rate of false negative under-the-bus warnings. False negative warnings are possible when a person falls while being occluded by another object or the last part seen by the sensor is close to another object and merges with it. Another possibility is that a person falls under the bus at the front door when the door is open. The system excludes these situations because people routinely disappear from the view of the sensor at that location when they enter the bus.
7.1.4. False Positive Warnings
Alert and imminent warnings. When we interviewed operators about the warning system, they stated that although the collision warning system is acceptable to them, they still would like a lower false warning rate. The false warning rates discussed below should therefore be considered in the context of an acceptable system, while recognizing that a reduction in the false warning rate is desirable.
We reviewed all alerts and imminent warnings in two runs and determined if they were true or false. One of the runs took place in California and the other in Pennsylvania, and together they were five hours long. We wanted to gather enough data to ensure that we observed the main categories of false warnings. We collected at least 30 warnings for each warning level and each side to ensure an upper limit of 10%, i.e., false warning categories that were not observed have a rate of less than 10% (90% confidence level). Table VII shows the absolute number of warnings, the relative number for each category (percentage of the total number of warnings), and the warning rates, for the left and right sides.
Table VII. True and false positive warnings
| ||Absolute||Relative (%)||Rate (1/h)|
The most common situations that cause true warnings are vehicles passing and fixed objects in the path of a turning bus. On the right side there are additional true warnings caused by pedestrians entering the bus or walking toward the bus when the bus has not yet come to a full stop.
A majority of the alerts are true alerts, whereas a majority of the imminent warnings are false positives. The most common reason for false imminent warnings is that the velocity estimation was incorrect, but as explained below this kind of error is not very serious.
Vegetation. The system cannot distinguish between vegetation (grass, bushes, and trees) and other fixed objects, but the threat posed by vegetation is much smaller than other objects because a bus can come in contact with grass, leaves, or small branches without being damaged. A warning triggered by vegetation can often be considered a nuisance warning. This is the least serious kind of system error because the system functions correctly, but the warning is considered a nuisance. Figure 16 shows a situation when the bus comes close to a bush and an imminent warning is triggered. On the right side, the four images from the four side cameras (see Figure 12) are shown. The bush can be seen in the upper right image, overlaid with a dotted and a thick white box, indicating an alert and an imminent warning for that object. Those boxes can also be seen in the bird's-eye view of the situation on the left. Notice that part of the bush extends over the curb. If an object is off the curb, the warning algorithm will give it a higher probability of collision than if it is on the curb.
Figure 16. Overhanging bush is close enough to trigger an alert (dotted box) and an imminent warning (thick white box).
Download figure to PowerPoint
False velocity estimation. The velocities of the objects are determined by our DATMO algorithm. Figure 9 shows the error distribution of the velocity estimation in the x and y direction from that report. The distribution is characterized by a Gaussian shape plus some additional outliers. The false velocities that give false warnings are from the tail of the Gaussian distribution or are outliers. An example of a case when a slightly incorrect velocity estimation leads to an alert is shown in Figure 17. The vehicle can be seen in the lower left image with a dotted box on it, indicating an alert. The dotted box also appears in the bird's-eye-view display.
The incorrect velocity estimation increases the probability of collision by enough to cross the warning threshold. It needs to be mentioned that this kind of error is not very serious because the danger level was only slightly overestimated. In most of the cases when an imminent warning was issued because of a false velocity estimation (such as Figure 17), the correct warning level would have been an alert.
No velocity information. Our DATMO needs some time to determine the velocity of an object after its initial detection (see the “velocity delay” discussion in Section 6.2). During that time, the system ignores the velocity of the object, i.e., sets it to zero. In some cases, this can lead to a false warning, especially if the object is in the path of the bus and moving away from the bus. This error is of medium seriousness because an object is present but the threat level is misjudged.
It is possible to avoid these false warnings by waiting until the velocity of the object has been established, but this would introduce latency and therefore false negative warnings.
Ground return. The laser scanners will see the ground if the bus rolls or pitches or if the ground in the vicinity of the bus slopes upward with respect to the plane on which the bus stands. Depending on the location of the ground seen by the sensor, the system might issue a warning, as shown, for example, in Figure 18.
In this case the bus is turning left. The laser scanner sees the ground and the system thinks an object is directly in the path of the bus and issues an imminent warning. In the left upper image, the ground return is indicated as a thick white (imminent warning) box and in the bird's-eye-view display it is the thick white box in the upper left corner. This is the most serious false positive, because a warning is issued when there is no threat whatsoever.
Other reasons for false positives. There are many other reasons for false positive warnings, which can vary greatly in their frequency from run to run. For example, in some runs a malfunction of the retraction mechanism misaligned the laser scanner and resulted in hundreds of false warnings. Some of the reasons were easily eliminated after their discovery, but they are listed here for completeness.
Retraction malfunction: When the laser scanner is retracted, a switch should signal this fact to the system. In some cases the switch malfunctioned and the sensor was retracted without the system knowing about it.
Dirt on the scanner: Dirt on the laser scanner appears as an object very close to the bus. This problem can be solved by discarding all scanner data points very close to the scanner itself.
Scanner sees the bus: The laser scanner can see parts of the bus if the scanner is slightly misaligned or if some parts of the bus protrude, such as an open door or a wheel when the bus is turning. This problem can be solved by excluding returns from certain areas, but this has the side effect that these areas are now blind spots.
Error in DATMO: There are many ways that our DATMO algorithm can make mistakes. The most common one was already mentioned above, this being the incorrect estimation of the velocity of an object.
Splashes: Splashes of water can be seen by the scanner and trigger a warning. In Figure 19, one can see the development of such a splash, indicated by a circle in the middle image. The outline of the bus is on the left in gray. The splashes are seen by the sensor for only about 0.2 s, but this is enough to be registered as an object and to trigger an alert.
Noise in turn rate: The turn rate of the bus is measured by a gyroscope, which has some noise, so there is a small chance that it gives an erroneous value. Very few cases were observed when these errors led to a false warning.
Dust: A cloud of dust can be produced by the wheels, appearing as an object to the system.
7.1.5. Contact and Under-the-bus Warnings
The rate of contact warnings is very low, about 0.4 warnings/h on the right side and 0.1 warnings/h on the left side. All the contact warnings we observed during normal operations were false positive warnings, i.e., no actual collisions occurred. These warnings were not directly related to our DATMO. They were caused by two different indicator failures. One was the indicator for the retraction of the laser scanner and the other was the bus-door-open indicator. Either indicator would have vetoed the collision warning.
For the under-the-bus warnings, we observed a false positive rate of 1.9 per hour. About half of those were caused by similar indicator failures to those for contact warnings. Most of the rest were cases when the system was not able to interpret a cluttered scene correctly. In two cases, the bus tires created a small dust cloud that disappeared after a short time and triggered a warning.
The total false positive warning rates of 0.5 contact warnings/h and 2 under-the-bus warnings/h are still too high because either warning requires drastic actions from the bus driver, namely stopping the bus and investigating what happened. We therefore did not activate and display these warnings for the operational testing in public service.
The fact that we did not observe any correct under-the-bus warning is no surprise, since people falling under the bus is an extremely rare event. The system would benefit from additional sensors that can positively identify that something is underneath the bus or that something hit the bus.