I have "dirty" LiDAR data containing first and last returns and also inevitably errors under and over the surface level. (screenshot)
I have SAGA, QGIS, ESRI and FME at hand, but no real method. What would be a good workflow to clean this data? Is there a full automated method or would I somehow be deleting manually?
You seem to have outliers:
For 'i', the option is to use a ground filter algorithm that can take into account 'negative blunders' to get a clean LiDAR ground point cloud. See the Multiscale Curvature Classification (MCC) algorithm from Evans and Hudak (2007). It is said on page 4:
Negative blunders are a common occurrence in LiDAR data, which may be caused by the scattering of the photons in a returned laser pulse. Scattering lengthens the time for an emitted laser pulse to return to the aircraft sensor, inflating the calculation of distance traveled, hence causing a measurement error where the surface elevation is erroneously recorded as being below the surrounding measurements. It should be noted that curvature classification approaches can potentially remove valid returns surrounding negative blunders, which can expand the edge artifact around a negative blunder to create a distinct “bomb crater” effect. To address negative blunders, Haugerud and Harding suggested setting the curvature tolerance parameter to four times the interpolated cell size and selecting returns exceeding this negative curvature threshold. However, it should be noted that under certain circumstances, returns that appear to be negative blunders can be in fact valid returns (e.g., sinkholes). Therefore, the preceding suggestion to remove potential negative blunders can be implemented as an optional last model loop to employ at the discretion of the user if needed.
Below there is a post with an example about using MCC-LIDAR:
Once you have an accurate LiDAR ground point cloud to make an accurate DEM, it is possible to normalize the point cloud, and exclude points which are beneath the DEM surface (the ones with negative values). Using the same approach, it is also possible to address point number 'iii' removing points above some fixed threshold. See, for example:
Then, it leaves us with 'ii', which is addressed by AlecZ's answer recommending lasnoise from LAStools. It will also handle 'iii', and perhaps part of 'i' as well (LAStools requires a license though). Other tools specifically created for checking/removing outliers were cited here: PDAL's filters.outlier tool in Charlie Parr's answer which has a detailed explanation about how the tool works, and with the advantage PDAL is a free software.
Then, what is left from the automated process (if any outlier) can be removed manually. For example:
Evans, Jeffrey S.; Hudak, Andrew T. 2007. A multiscale curvature algorithm for classifying discrete return LiDAR in forested environments. IEEE Transactions on Geoscience and Remote Sensing. 45(4): 1029-1038.
I'll recommend PDAL the point data abstratction library. I've had good success using PDAL for a similar filtering problem. I like PDAL because it is open source, provides Python support, and makes it easy for me to reproduce the processing and keep track of my filtering parameters. I also like it because it has 'pipelines' where you can chain together several steps (e.g. crop then filter then export) and execute them at once. Note that if you have really, really large point clouds PDAL might not be as speedy as some other solutions (LASTools, QTM, etc.).
You could address the issue of outlying points with a PDAL pipeline similar to the following:
{
"pipeline": [
"input_utm.las",
{
"type":"filters.crop",
"bounds":"([401900,415650],[7609100,7620200])"
},
{
"type":"filters.outlier",
"method":"statistical",
"mean_k":12,
"multiplier":2.0
},
{
"type":"filters.range",
"limits":"Classification![7:7]"
},
{
"filename":"output.tif",
"resolution":1.0,
"output_type":"mean",
"radius":3.0,
"bounds":"([401900,415650],[7609100,7620200])",
"type": "writers.gdal"
}
]
}
This pipeline reads in an LAS, crops it to a specified UTM extent, then performs a filter which flags all outlying points, then performs a second filter which retains only non-outlying points (i.e. the Classification flag != 7), then exports to a 1 m resolution GeoTIFF. The statistical filter is performing a nearest neighbor mean distance computation to test whether or not a point is 'too far' from its neighbors and therefore an outlier.
From the documentation:
Since OP did not limit solutions to open-source, I'd suggest Quick Terrain Modeler (QT Modeler). It does require a license. Load the point cloud in QT, and you essentially tilt it to get the profile view you want, rubber-band the cluster you want to remove, and just hit delete.
I have had luck simply using a focal variance on an interpolated raster. You then assign the variance values to your points and use a threshold to remove locally high variances, representing large departures from the local kernel estimate.
You do have to make sure that the resolution of the interpolated surface is a small enough grain that it captures local variation at a point(s) level. The size of the kernel will have an effect as well but for single outliers a 3x3 window should suffice. You may loose a few additional points hear-and-there but, with lidar you have ample data to spare.
Lastools provides exactly what you need - automated scripts that will remove all these points for you. There is a licensing cost for that, however, but if this is a process you want to quickly do as a regular task, using the lasnoise script from their toolset is a perfect option.
As @Andre Silva noted, ArcGIS has a las toolset, which you can use after running the Create LAS Dataset geoprocessing tool. From there, you can go in manually to reclassify or delete these noise points. The drawback is that it's not as intuitive or effective a process as QT Modeler (suggested by @auslander), probably the best program for visualizing/analyzing/manipulating las files manually, and with a license cost as well. ArcMap will limit the number of visible points when editing your point cloud, meaning that you will likely have to zoom in to areas with noise, remove or reclassify them, and then move through as a part of a manual cleanup process. But this will get the job done.
As Andre Silva has said, MCC-LIDAR is a good option to extract the ground points but from my experience, it will struggle if you have a very big pointcloud (500 million points or even less). In other words, it will return an error and won't run the algorithm, even if you change the settings (scale and curvature parameters). Also, from my experience, it keeps some of the "negative blunders" in the data.
My alternative for that is to invert the pointcloud (the points below ground will go up and the above ground will go down). To get this, I load the data into R and invert the height, then run MCC-LIDAR and re-invert the data. You could probably do this in QGIS or ArcGIS but depending on the size of your dataset, it could take a while to do.
The PDAL tool ground is also a good option as it works better with larger datasets but, again, some of the points bellow ground will still be kept. Inverting the dataset will again help to solve this issue.
For the points above ground, my best approach is a manual cleaning and the best open source tool I've found to do so is within CloudCompare. You'll choose Segment in the top bar menu and you can either remove the points selected or all the others. I've used LAStools before (lasview tool) for this but the way the 3D interface works is not as user friendly.
I'm a technical support from GreenValley International, the Outlier Removal tool in our flagship software LiDAR360 can be used to remove these errors as much as possible and therefore improve the data quality.
The algorithm will first search for each point's neighboring points within a user-defined neighborhood, and calculate the average distance from the point to its neighboring points. Then, the mean and standard deviation of these average distances for all points are calculated. If the average distance of a point to its neighbors is larger than maximum distance (maximum distance = mean + n * standard deviation, where n is a user-defined multiple number), it will be considered as a outlier and be removed from the original point cloud.
As an Open Source option, 3D Forest has some nice tools to filter automatically, as well as manual tools to clean point clouds. You might have to try with diferent filter parameters to get the result you need. Even though it´s oriented to forest point clouds, many tools are usefull in any point cloud.
