For autonomous vehicles and ADAS, lidar is crucial in providing a 3D map of all objects that surround the vehicle regardless of external lighting conditions. The 3D map is updated hundreds of times per second, enabling a vehicle to estimate its position, in real time, in relation to its surroundings.
At present, lidars do not have a standardized measure for describing their performance levels, meaning there is no widely accepted protocol for comparisons between differing lidar solutions. Despite some comparisons being possible through the use of a manufacturer’s own specifications, these could also be viewed as inaccurate due to varying performance metrics and confidential information.
To address the problem, Dr Paul McManamon of Exciting Technology formed a group in partnership with SPIE, the international society for optics and photonics (SPIE). The project is to run for three years and aims to develop tests and performance standards for lidars in AVs and ADAS.
During the project’s first year, tests were headed up by Dr Jeremy Bos, an associate professor at Michigan Technological University (MTU), supported by PhD student Zach Jeffries. Other authors included Charles Kershner from the National Geospatial-Intelligence Agency, who was responsible for setting up a ground truth Riegl lidar for the test, and Akhil Kurup from MTU.
The findings of the first-year test were recently published in an open-access paper in Optical Engineering, in addition to the team explaining what it plans to do over the remainder of the project.
“Lidar engineers make design trade-offs to gain competitive advantages in performance and cost in what is a rapidly growing, highly competitive market,” commented Jeffries. “Some of these trade-offs include operating wavelength (typically between 850Nm and 1,600Nm), range measurement based on either direct detection/time-of-flight (ToF) or coherent techniques, beam steering solutions (mechanically rotating components, MEMS mirrors, micro lenses), and laser source type [vertical cavity surface-emitting lasers (VCSELs), edge-emitting diodes]. These design choices have trade-offs of their own, with differences in scan patterns, sampling frequency, achievable ranges, susceptibility to interference from other lidars, etc.”
During these tests, the team’s objective was to evaluate the range, accuracy and precision of eight automotive-grade lidars using a survey-grade lidar as a reference. The team set up an array of targets along a 200m path in an open field in Kissimmee, Florida. A key aspect of the targets which made the tests stand out from previously conducted studies was that they were near-perfect matte surfaces with a calibrated 10% reflectivity across a wide spectrum. During testing the team also measured the ability of the lidars to detect a target among highly reflective road signs.
Generally, the subsequent test results were consistent and in keeping with the values written in the manufacturer’s datasheets. However, despite recording a mean precision of 2.9cm across all the devices that the team tested, the distribution of the measured values was not Gaussian.
The team stated there was a non-negligible probability for these devices to report very imprecise values (an error greater than 10cm). In some cases the measured range deviated from the real value by as much as 20cm. Additionally, other important results showed that the reflective road signs impaired the target detection performance of the lidars.
“The advertised range performance of lidars pertains to very specific conditions, and performance degrades significantly in the presence of a highly reflective adjacent object,” said Bos.
The first-round tests provided the researchers with valuable insights into the performance differences between different lidars. The results suggested that the metrics reported by each respective manufacturer were not reliable.
“The first-year tests were the simplest of them,” explained Bos. “In the second year, we will duplicate these tests for the characterized lidars while introducing confusion resulting from other automotive lidars approaching from the opposite direction. Additionally, we will measure the eye safety of the lidars. Finally, in the third year, we will include weather effects as a culmination of the complexity build-up.”
