LiDAR DEM Quality Control Report. The accuracy of a LiDAR DEM is estimated by determining the root mean square error (RMSE). RMSE is the square root of the average of the set of squared differences between dataset co-ordinate values and co-ordinate values from an independent source of higher accuracy for identical points. If those differences are normally distributed and average zero, 95 percent of any sufficiently large sample should be less than 1.96 times the RMSE. Therefore 15-centimeter RMSE is often referred to as "30-centimeter accuracy at the 95-percent confidence level". Following that convention, the vertical accuracy of any DEM is defined as 1.96 times the RMSE of linearly interpolated elevations in the DEM, as compared with known elevations from high-accuracy test points. DEMs should have a maximum RMSE of 15 centimeters, which is roughly equivalent to 1-foot accuracy. Field verification of the vertical accuracy of this DEM to ensure that the 15-centimeter RMSE requirement was satisfied for all major vegetation categories that were predominate
a) Bare-earth and low grass (plowed fields, lawns, golf courses);
b) High grass and crops (hay fields, cornfields, wheat fields);
c) Brush lands and low trees (chaparrals, mesquite, mangrove swamps);
d) Fully covered by trees (hardwoods, evergreens, mixed forests); and
e) Urban areas (high, dense man-made structures). An even distribution of sample points throughout each category area evaluated was collected and not grouped in a small subarea. The RMSE calculated from a sample of test points is not the RMSE of the DEM. The calculated value may be higher or it may be lower than that of the DEM. Confidence in the calculated value increases with the number of test points. If the errors (lack of accuracy) associated with the DEM are normally distributed and unbiased, the confidence in the calculated RMSE can be determined as a function of sample size. Similarly, the sample RMSE necessary to obtain 95-percent confidence that the DEM RMSE is less than 15 centimeters can also be
determined as a function of sample size. For each major vegetation category, a sample of points was tested to show the test points have an RMSE less than where n is the number of test points in the sample. A minimum of 20 test points for each major vegetation category was identified. Therefore, a minimum of 100 test points was selected for the five major vegetation categories. The test points were to be selected in areas to evaluate DEM accuracy under trees and in vegetation representative of the study area. The PDOP during the LiDAR data collection was consistently less than 3.0 and was determined to be of no issue. Test points on sloping or irregular terrain would be unreasonably affected by the linear interpolation of test points from surrounding DEM points and, therefore, were not selected. Test points were collected by RTK (Real-Time Kinematic) GPS techniques. Three thousand Two Hundred and Sixty points were collected in total covering each of the five main categories of ground cover in the survey areas. Furthermore, six of the forty-eight control monuments falling within the project area and installed as part of the survey network were used as a further check. All RMSE calculations were performed on the bare-earth, orthometric surface. Results The comparisons between each validation point and the LiDAR DEM are shown in Appendix A. The comparisons between each control point and the LiDAR DEM are shown in Appendix B. The RMSE was determined for the project area. US Survey Feet Meters Average dz 0.144 0.044 Average magnitude 0.332 0.101 Root mean square 0.395 0.120 Std deviation 0.369 0.112 US Survey Feet Meters Average dz 0.246 0.075 Average magnitude 0.451 0.137 Root mean square 0.571 0.174 Std deviation 0.520 0.158 The favorable result of the DEM comparison to the validation points provides an overall confidence that the LiDAR system was operating properly during data collection.
The scattering of the test points over the project area assists in this determination. Those points in both the control and validation sets marked as outside are such as they fall outside of a predetermined maximum triangle size or are outside of the project area. Therefore, there are an insufficient number of LiDAR points hitting the ground in the immediate vicinity of these test points. Two test points and four control points were removed from the report as they fall on steeply sloping triangles. Hence, any attempt to assign a value from the triangulated surface will result in erroneous values and so these points are excluded from the RMSE calculation. Due to the nature of the area and in-definite spot of each individual LiDAR point, an RMSEh value was not reported. Any particular point cannot be tested. However, accuracy
statements can be made about the performance of the ABGPS, IMU and LiDAR sensor. The ABGPS data are quality controlled by comparing multiple solutions from multiple base stations. On this project, these solutions all agreed to better than 5 cm horizontally. The IMU sensor combines the post-processed GPS data with the raw inertial data to produce a best estimate of trajectory. Automated quality control checks will not allow the IMU solution to be of less accuracy than the provided input from the GPS solution. The altitude of the sensor on this project was 1220 meters (4003 US Survey Feet) AGL providing a spot size of 37 cm (1.2') in diameter. Each return is located somewhere within the spot on the ground, meaning the location of the point is located within 17.5 cm of the center of the spot. The stated horizontal accuracy of the system is 1/1000 of the altitude. On this project, the combination of all the errors from all the components of the sensor is much less than the stated accuracy. Conclusions. The final DEM generated for this project is accurate in all types of vegetation and ground cover with the exception of those areas of high grasses. High grass areas are expected to provide some discrepancies due to the density of the grasses and the inability to penetrate these areas sufficiently. The
accuracy of the DEM on bare-earth and low grasses, and the scattering of those points over the study area, provides proof that the LiDAR system that collected the DEM was operating correctly. Tested 0.235 meters consolidated vertical accuracy at ninety-five percent confidence level in open terrain and grassy areas using RMSE (z) x 1.9600. Expected horizontal accuracy of elevation products as determined from system studies and other methods is 1/1000th of the flight height, which in the instance of this particular project was 1220m (4002.6US survey feet) AGL, giving a horizontal tolerance of less than 1.22m (4.0 US survey feet). Respectfully Submitted, MD Atlantic Technologies, Inc. Darrick L. Wagg, P.Geo. 03Jun2004
