Surveying payload Accuracy report
Last updated
Was this helpful?
Last updated
Was this helpful?
Firstly, it is important to understand the differentiation between accuracy and precision.
Accuracy refers to the geographical precision of a tool. This measures how closely the LiDAR measurements match real-world values. For example, imagine that you are scanning a wall. If your LiDAR point cloud (a digital version of the wall) produces measurements and distances that match the real-world wall, then the accuracy is high. We measure accuracy in terms of distance errors, (i.e. centimeters or inches). This accuracy measurement is crucial for applications that require measurements as close to reality as possible.
On the other hand, precision refers to the replicable nature of a measurement. How consistently can it make a measurement, and how true-to-reality is that measurement? A ruler can measure 30 cm very precisely every time because its measurement is clearly defined. When it comes to LiDAR for drones, precision is defined by the thickness of the point cloud. In the example of scanning a wall, the point cloud for a precise laser scan will be very thin - matching the wall. If there are lots of scattered points (called “noise”), then that point cloud is not very precise.
So, to understand the relationship between accuracy and precision, you can refer to these 4 diagrams of a square. When there is high accuracy (the points match the location of the wall) but precision is low (there is noise in the point cloud), you have the top left version, that loosely matches the real structure of the square. Alternatively, when the accuracy and precision are both close to reality, you can see that there is little noise in the point cloud, and the points all closely follow the outline of the square.
The new payload for the Elios 3 is the Ouster OS0 128 Rev 7. This payload has greater accuracy and precision than previous iterations. As part of our testing, we plotted captured points on a bell curve, looking at the standard deviation. This is used to quantify the level of noise or uncertainty of a LiDAR measurement on a planar surface. A higher standard deviation indicates greater variability in the point cloud, and thus lower precision. We have found that the precision of the new payload is accurate to +/- 6 mm at 1 sigma. This means that 66% of the points measured are accurate to within 6 mm. 95% of the points are accurate to within 12 mm (2 sigma). This means that almost all of the points in a point cloud are, on average, as accurate as 6mm, and thus the Rev 7 payload is precise to within 1 centimeter of reality.
The LiDAR data captured with the Surveying Payload is processed with FARO Connect, one of the leading LiDAR data processing software programs.
Global Accuracy Testing and Results for the Surveying Payload
The way we test accuracy and precision in a point cloud is by determining the level of drift. Drift is a key metric used to express the accuracy of a mapping system. The term is used in 3D modeling to describe the cumulative decrease in accuracy over the duration of a capture. Accuracy cannot easily be expressed in absolute values unless you have a clear system of reference. This is why surveyors use ground control points or GNSS to tether their point clouds to real-world coordinate systems or reference points. Without GNSS or ground control points (GCPs), the absolute error of a point cloud typically expands as the asset/area being surveyed increases in size.
To explain this with numbers, you can expect the error on a 30-meter (98-foot) measurement to be smaller than the error on a 300-meter (984-foot) distance measurement because a mobile scanner moving through the space will accumulate errors on top of previous errors. This accumulation of errors over distance is what we call drift, and represents a percentage of the traveled distance during data collection - for example, a 1% drift on a 300-meter distance corresponds to a 3-meter error compared to reality.
Understanding factors that can affect global accuracy
Global accuracy is impacted by the size and characteristics of the area being surveyed, as well as surveying method.
When it comes to surveying complex, confined spaces where the Elios 3 is at work, there can be added challenges (and factors that increase the drift) if an environment has little variation (also known as being homogenous). This is typical in assets like pipes, chimneys, and tunnels. They can be incredibly complex to survey due to the homogenous nature of the space. This is because LiDAR relies on detecting and measuring features on surfaces, such as corners, edges, or texture variations to create 3D point clouds. When an environment is symmetrical, there are fewer feature points, making it more difficult for the LiDAR to identify and track reference points for accurate measurements.
It should also be noted that the method of data collection will also affect the quality of results. For example, flying the drone too fast can limit successful data capture, while avoiding collisions optimizes data collection. Further details on this are available via Flyability and FARO’s training resources.
Legend: When there are fewer geometric features, as in highly symmetrical environments, it is harder for the LiDAR to detect key features of reference that enable it to correctly interpret its surroundings, causing it to accumulate drift.
Bearing all of these factors in mind, we tested the Elios 3’s Surveying Payload in several environments with varying degrees of symmetry to assess how it handles these scenarios.
Structured environments are ones with little to no symmetry as well as feature points - such as buildings, stockpiles, and containment areas. They also have a diameter or distance between walls that is over 2 meters wide (6.5 feet).
In our test, we went to the basement of a factory. We used data from a Terrestrial Laser Scanner (Reigl VZ400) to scan the entire area to produce a highly accurate and precise ground truth model of the test environment. From this scan, we identified a 15x15 meter section that we used as the take-off and landing area. We would use this area to align multiple point clouds through a processing setting called Iterative Closest Point (ICP). We conducted multiple flights with the Elios 3 standard configuration (Rev 6.2) and the Elios 3 Surveying payload (Rev 7). Each Elios 3 flight was aligned to the ICP area and the computer transformation was then applied to each individual Elios point cloud so that each flight was registered to the 15x15 meter ICP location. The reference centroids from the TLS data were recorded and compared to the registered Elios 3 detected target centroids from each individual flight with the new transformations applied.
The results, as shown in this table, demonstrated significant improvements in the Surveying payload. The previous LiDAR payload (The Rev 6.2) had 0.5% drift. The new Rev 7 model is 4x more accurate, with 4x less drift. The drift reduced to just 0.16%, showing a high degree of accuracy across the entire space (global accuracy).
The next testing area was a partially symmetrical environment. These environments have more than 2 meters of width and height and have regular geometric features or clear bends at intervals of max 30-50 meters.
In this case, we conducted tests in 2 locations. The first environment was a corridor consisting of an electrical conduit, pipe racks, and various concrete structures. The second was a 200m tunnel with slight curves.
In-depth analysis of nominally symmetrical environment results
The first test took place in the corridor where there were more features that helped the LiDAR scan reduce drift. These features included pipes, racks, and an electrical conduit along with the overall structure being over 2 meters in diameter. The result is that there is a 5-10 times improvement in drift for the new Rev 7 payload compared to the Rev 6.2. This highlights just how critical geometric features are in reducing overall drift for 3D digitalization.
The second accuracy test took place in the sewer tunnel. In this case, 3 scans were conducted with the Rev 7 LiDAR payload, and processed with FARO Connect. The scans were then compared with 3 identical flights with the LiDAR Rev 6.2 payload (the standard configuration for the Elios 3). Both data sets were aligned in their respective projects with the ICP method using points around the entrance that had been ground-truth captured with a TLS, along with survey targets every 25 meters inside the tunnel. However, while data from the Rev 7 was processed in FARO Connect, the Rev 6.2 data was viewed with FlyAware, the standard inspection software for the Elios 3 and not part of the Surveying Payload package.
This cross-section from the beginning of the tunnel shows colored point clouds from the LiDAR Surveying Payload(Rev 7), the Terrestrial Laser Scanner as a control dataset, and the standard Elios 3 LiDAR, the Rev 6.2. This was the area used to align the different point clouds with the ICP settings in FARO Connect.
This is the sewer cross-section at the end of the flight. As you can see, the green Rev 6.2 has significantly more drift, reaching 1.4%, compared to the Surveying Payload Rev 7 and FARO Connect, which only has 0.19% drift.
Here you can see where the cross-sections were taken in the overall scan.
Overall, the average 0.39 meters difference from reality by the Rev 7 payload highlights the improved robustness of the payload in symmetrical environments compared to the Rev 6.2. Its global accuracy in this project was 5-8 times better than the standard LiDAR payload.
The clear superiority of the Rev 7 in these environments thus makes it the preferred payload for surveying environments with nominal symmetry.
The Surveying payload was tested in increasingly challenging environments. A challenging symmetrical environment is one with light geometric features or texture in prolonged straight areas (greater than 30-50 meters), as well as a diameter greater than 2 meters. Examples of environments with these features could include tunnels, stacks, and shafts.
In this testing environment, the Rev 7 payload was compared to the Rev 6.2 in a sewer with greater than 2 meter diameter. The only geometric features were walkways, a gully, and shotcrete with textured surfaces.
In comparing the flights, the Elios 3 Rev 6.2 payload, processed with FlyAware, had 2-5% drift. The Elios 3 Surveying Payload, using FARO Connect, achieved results that were 2.5x times better. The drift was limited to just 0.5 - 2% in various sections of the tunnel, resulting in a 50-80% convergence success rate.
In environments with very few geometric features that make drift more likely, the Rev 7 payload is still achieving improved results compared to the Rev 6.2 thanks to the improved LiDAR capabilities as well as processing with FARO Connect.
As part of final testing, the Rev 7 Surveying Payload was then used to scan another sewer environment. It was considered to be very challenging because the diameter was less than 2 meters (0.8m to be exact). In addition, there were few geometric features or textures in straight sections.
A narrow sewer tunnel was used for testing. This tunnel had curves, bends, and deviations in its trajectory after 20-30 meters. The only geometrical features in the straight sections were bricks or textured surfaces, manholes, and inlet/outlet pipes.
The Surveying Payload and FARO Connect were still capable of acceptably accurate results, with a drift of 2-5 % (measuring a convergence success rate of 50-80%). For comparison, the Rev 6 standard configuration with Flyaware had over 5% drift in this difficult environment.
This table summarizes the findings of these accuracy tests, with comparisons between the standard Elios 3 6.2 Rev data and the Elios 3 Surveying Payload with FARO Connect.
Elios 3 & FlyAware
Surveying Payload & FARO Connect
Structured environments
Buildings, stockpiles, containment areas
Little to no symmetry
Geometric features
Diameter/distance between walls >2m meters (6.5 feet)
1x
0.5-1% drift
5-10x
~0.1-0.2%
Nominal symmetric environments
Tunnels, stacks, shafts
Diameter >2m (6.5 feet)
Regular geometric features
1x
~2% drift
5-10x
~0.25-0.5%
Challenging symmetrical environments
Tunnels, stacks, shafts
Diameter >2m (6.5 feet)
Light geometric features
1x
2-5% drift
2-5x
0.5-2%
(50-80% success rate)
Very challenging symmetrical environments
Tunnels, pipes, stacks, shafts
Diameter <2m (6.5 feet)
Light geometric features
1x
5+% drift
1-2x
2-5%
(50-80% success rate)
The new Rev 7 Surveying payload has achieved stunning results even in complex surveying environments. In combination with FARO Connect, it is capable of achieving sub-centimeter accuracy over large survey areas.
These results will appeal to surveyors and inspectors working not only in wastewater management but also in mining, manufacturing industries such as cement, and industry standardizing bodies.
With overall precision to within +/- 6mm and replicable accuracy results, the new Surveying Payload for the Elios 3 is the ideal solution for those looking to gather data in complex and potentially hazardous environments - be they confined spaces or larger structures - need an extra level of accuracy.
