Assessing the global accuracy of the Elios 3 Surveying Payload
An in-depth accuracy report analyzing test results of the Surveying Payload in various environments to determine its level of global accuracy.
Since its release in 2022, the Elios 3 has become a key instrument in the surveyor toolbox for capturing LiDAR data in areas where it was never possible to capture data before. With the growing need for better and more efficient data capture, sectors like mining, construction, and infrastructure management are turning to the Elios 3 to conduct safer inspections and surveys with greater data coverage.
Now, with the introduction of the Elios 3’s new Surveying Payload, the capabilities of this drone have been augmented. This report will detail the results of testing the new payload’s accuracy and demonstrate exactly how it has added to the Elios 3’s data collection workflows.
1. Defining Accuracy and Precision
Firstly, it is important to understand the differentiation between accuracy and precision.
Accuracy refers to the geographical precision of a tool. This measures how closely the LiDAR measurements match real-world values. For example, imagine that you are scanning a wall. If your LiDAR point cloud (a digital version of the wall) produces measurements and distances that match the real-world wall, then the accuracy is high. We measure accuracy in terms of distance errors, (i.e. centimeters or inches). This accuracy measurement is crucial for applications that require measurements as close to reality as possible.
Legend: In the high-accuracy versions on the left, you can see that the points match the location of the wall.
On the other hand, precision refers to the replicable nature of a measurement. How consistently can it make a measurement, and how true-to-reality is that measurement? A ruler can measure 30 cm very precisely every time because its measurement is clearly defined. When it comes to LiDAR for drones, precision is defined by the thickness of the point cloud. In the example of scanning a wall, the point cloud for a precise laser scan will be very thin - matching the wall. If there are lots of scattered points (called “noise”), then that point cloud is not very precise.
Legend: A precise point cloud, as shown on the left, has little “noise” - the points closely match the shape of the real-world object.
So, to understand the relationship between accuracy and precision, you can refer to these 4 diagrams of a square. When there is high accuracy (the points match the location of the wall) but precision is low (there is noise in the point cloud), you have the top left version, that loosely matches the real structure of the square. Alternatively, when the accuracy and precision are both close to reality, you can see that there is little noise in the point cloud, and the points all closely follow the outline of the square.
2. The Accuracy and Precision of the New Surveying Payload
The new payload for the Elios 3 is the Ouster OS0 128 Rev 7. This payload has greater accuracy and precision than previous iterations. As part of our testing, we plotted captured points on a bell curve, looking at the standard deviation. This is used to quantify the level of noise or uncertainty of a LiDAR measurement on a planar surface. A higher standard deviation indicates greater variability in the point cloud, and thus lower precision. We have found that the precision of the new payload is accurate to +/- 6 mm at 1 sigma. This means that 66% of the points measured are accurate to within 6 mm. 95% of the points are accurate to within 12 mm (2 sigma). This means that almost all of the points in a point cloud are, on average, as accurate as 6mm, and thus the Rev 7 payload is precise to within 1 centimeter of reality.
The LiDAR data captured with the Surveying Payload is processed with FARO Connect, one of the leading LiDAR data processing software programs.
Legend: On the left, there is a wall that we scanned to get the points used for the standard deviation calculation. As you can see in the bull curve, 66% of points with the Surveying payload fall +/- 6mm of reality.
Global Accuracy Testing and Results for the Surveying Payload
The way we test accuracy and precision in a point cloud is by determining the level of drift. Drift is a key metric used to express the accuracy of a mapping system. The term is used in 3D modeling to describe the cumulative decrease in accuracy over the duration of a capture. Accuracy cannot easily be expressed in absolute values unless you have a clear system of reference. This is why surveyors use ground control points or GNSS to tether their point clouds to real-world coordinate systems or reference points. Without GNSS or ground control points (GCPs), the absolute error of a point cloud typically expands as the asset/area being surveyed increases in size.
To explain this with numbers, you can expect the error on a 30-meter (98-foot) measurement to be smaller than the error on a 300-meter (984-foot) distance measurement because a mobile scanner moving through the space will accumulate errors on top of previous errors. This accumulation of errors over distance is what we call drift, and represents a percentage of the traveled distance during data collection - for example, a 1% drift on a 300-meter distance corresponds to a 3-meter error compared to reality.
Understanding factors that can affect global accuracy
Global accuracy is impacted by the size and characteristics of the area being surveyed, as well as surveying method.
When it comes to surveying complex, confined spaces where the Elios 3 is at work, there can be added challenges (and factors that increase the drift) if an environment has little variation (also known as being homogenous). This is typical in assets like pipes, chimneys, and tunnels. They can be incredibly complex to survey due to the homogenous nature of the space. This is because LiDAR relies on detecting and measuring features on surfaces, such as corners, edges, or texture variations to create 3D point clouds. When an environment is symmetrical, there are fewer feature points, making it more difficult for the LiDAR to identify and track reference points for accurate measurements.
It should also be noted that the method of data collection will also affect the quality of results. For example, flying the drone too fast can limit successful data capture, while avoiding collisions optimizes data collection. Further details on this are available via Flyability and FARO’s training resources.
Legend: When there are fewer geometric features, as in highly symmetrical environments, it is harder for the LiDAR to detect key features of reference that enable it to correctly interpret its surroundings, causing it to accumulate drift.
Bearing all of these factors in mind, we tested the Elios 3’s Surveying Payload in several environments with varying degrees of symmetry to assess how it handles these scenarios.
2.1 Accuracy in Structured Environments
Structured environments are ones with little to no symmetry as well as feature points - such as buildings, stockpiles, and containment areas. They also have a diameter or distance between walls that is over 2 meters wide (6.5 feet).
In our test, we went to the basement of a factory. We used data from a Terrestrial Laser Scanner (Reigl VZ400) to scan the entire area to produce a highly accurate and precise ground truth model of the test environment. From this scan, we identified a 15x15 meter section that we used as the take-off and landing area. We would use this area to align multiple point clouds through a processing setting called Iterative Closest Point (ICP). We conducted multiple flights with the Elios 3 standard configuration (Rev 6.2) and the Elios 3 Surveying payload (Rev 7). Each Elios 3 flight was aligned to the ICP area and the computer transformation was then applied to each individual Elios point cloud so that each flight was registered to the 15x15 meter ICP location. The reference centroids from the TLS data were recorded and compared to the registered Elios 3 detected target centroids from each individual flight with the new transformations applied.
The results, as shown in this table, demonstrated significant improvements in the Surveying payload. The previous LiDAR payload (The Rev 6.2) had 0.5% drift. The new Rev 7 model is 4x more accurate, with 4x less drift. The drift reduced to just 0.16%, showing a high degree of accuracy across the entire space (global accuracy).
Legend: This vertical cross-section through the floor and ceiling of the basement shows the Rev 7 and FARO Connect results closely matching the TLS point cloud (in red) compared to the Rev 6.2 results, shown in green.
2.2 Accuracy in Nominally Symmetrical Environments
The next testing area was a partially symmetrical environment. These environments have more than 2 meters of width and height and have regular geometric features or clear bends at intervals of max 30-50 meters.
In this case, we conducted tests in 2 locations. The first environment was a corridor consisting of an electrical conduit, pipe racks, and various concrete structures. The second was a 200m tunnel with slight curves.
In-depth analysis of nominally symmetrical environment results
Legend: The variety of geometric features in this corridor meant that drift was reduced.
The first test took place in the corridor where there were more features that helped the LiDAR scan reduce drift. These features included pipes, racks, and an electrical conduit along with the overall structure being over 2 meters in diameter. The result is that there is a 5-10 times improvement in drift for the new Rev 7 payload compared to the Rev 6.2. This highlights just how critical geometric features are in reducing overall drift for 3D digitalization.
The second accuracy test took place in the sewer tunnel. In this case, 3 scans were conducted with the Rev 7 LiDAR payload, and processed with FARO Connect. The scans were then compared with 3 identical flights with the LiDAR Rev 6.2 payload (the standard configuration for the Elios 3). Both data sets were aligned in their respective projects with the ICP method using points around the entrance that had been ground-truth captured with a TLS, along with survey targets every 25 meters inside the tunnel. However, while data from the Rev 7 was processed in FARO Connect, the Rev 6.2 data was viewed with FlyAware, the standard inspection software for the Elios 3 and not part of the Surveying Payload package.
This cross-section from the beginning of the tunnel shows colored point clouds from the LiDAR Surveying Payload(Rev 7), the Terrestrial Laser Scanner as a control dataset, and the standard Elios 3 LiDAR, the Rev 6.2. This was the area used to align the different point clouds with the ICP settings in FARO Connect.
This is the sewer cross-section at the end of the flight. As you can see, the green Rev 6.2 has significantly more drift, reaching 1.4%, compared to the Surveying Payload Rev 7 and FARO Connect, which only has 0.19% drift.
Here you can see where the cross-sections were taken in the overall scan.
Overall, the average 0.39 meters difference from reality by the Rev 7 payload highlights the improved robustness of the payload in symmetrical environments compared to the Rev 6.2. Its global accuracy in this project was 5-8 times better than the standard LiDAR payload.
The clear superiority of the Rev 7 in these environments thus makes it the preferred payload for surveying environments with nominal symmetry.
2.3 Accuracy in challenging symmetrical environments
The Surveying payload was tested in increasingly challenging environments. A challenging symmetrical environment is one with light geometric features or texture in prolonged straight areas (greater than 30-50 meters), as well as a diameter greater than 2 meters. Examples of environments with these features could include tunnels, stacks, and shafts.
Legend: The variety of geometric features in this corridor meant that drift was reduced This tunnel has little to no clear geometric features to help reduce drift in a LiDAR scan.
In this testing environment, the Rev 7 payload was compared to the Rev 6.2 in a sewer with greater than 2 meter diameter. The only geometric features were walkways, a gully, and shotcrete with textured surfaces.
In comparing the flights, the Elios 3 Rev 6.2 payload, processed with FlyAware, had 2-5% drift. The Elios 3 Surveying Payload, using FARO Connect, achieved results that were 2.5x times better. The drift was limited to just 0.5 - 2% in various sections of the tunnel, resulting in a 50-80% convergence success rate.
In environments with very few geometric features that make drift more likely, the Rev 7 payload is still achieving improved results compared to the Rev 6.2 thanks to the improved LiDAR capabilities as well as processing with FARO Connect.
2.4 Accuracy in very challenging symmetrical environments
As part of final testing, the Rev 7 Surveying Payload was then used to scan another sewer environment. It was considered to be very challenging because the diameter was less than 2 meters (0.8m to be exact). In addition, there were few geometric features or textures in straight sections.
A narrow sewer tunnel was used for testing. This tunnel had curves, bends, and deviations in its trajectory after 20-30 meters. The only geometrical features in the straight sections were bricks or textured surfaces, manholes, and inlet/outlet pipes.
The Surveying Payload and FARO Connect were still capable of acceptably accurate results, with a drift of 2-5 % (measuring a convergence success rate of 50-80%). For comparison, the Rev 6 standard configuration with Flyaware had over 5% drift in this difficult environment.
Legend: This narrow sewer had a slightly sloped angle
Legend: With greater geometrical features in a similar environment, it may be possible to reduce the drift. However, considering the challenge of this location, the Surveying Payload’s Rev 7 handled the survey very well with better results than the Rev 6.2.
Summary of findings and conclusion
This table summarizes the findings of these accuracy tests, with comparisons between the standard Elios 3 6.2 Rev data and the Elios 3 Surveying Payload with FARO Connect.
| Environment | Configuration | ||
| Elios 3 & FlyAware | Surveying Payload & FARO Connect | ||
|
Structured environments |
|
1x 0.5-1% drift |
5-10x ~0.1-0.2% |
|
Nominal symmetric environments |
|
1x ~2% drift |
5-10x ~0.25-0.5% |
|
Challenging symmetrical environments |
|
1x 2-5% drift |
2-5x 0.5-2% (50-80% success rate) |
|
Very challenging symmetrical environments |
|
1x 5+% drift |
1-2x 2-5% (50-80% success rate) |
The new Rev 7 Surveying payload has achieved stunning results even in complex surveying environments. In combination with FARO Connect, it is capable of achieving sub-centimeter accuracy over large survey areas.
These results will appeal to surveyors and inspectors working not only in wastewater management but also in mining, manufacturing industries such as cement, and industry standardizing bodies.
With overall precision to within +/- 6mm and replicable accuracy results, the new Surveying Payload for the Elios 3 is the ideal solution for those looking to gather data in complex and potentially hazardous environments - be they confined spaces or larger structures - need an extra level of accuracy.