Cleaning blocks pattern from urban LIDAR (blocks uplifted from streets)

Question

We have a 1-meter LIDAR DEM from a city.

A small subset can be downloaded from this link:

This screenshot shows raw DEM with gray palette (darker belts are streets, and greyish and whitish rectangles are blocks):

This correspond to a place in Santo Domingo city, which can be seen in this Google screenshot:

On average, blocks are "uplifted" ca. 2 meters from streets, which is not true. We want to have a clean DEM to generate stream network and topographic wetness index (TWI). With the DEM supplied (we don't have the original bands from laser scanner), the hydrgraphic network seemed to follow a rectangular layout, and TWI resulted in blocks pattern. These pictures show the results:

This is the stream network result, generated with r.watershed in Grass GIS:

And this is TWI result, generated with SAGA:

We tried some procedures to solve this inaccuracy without success:

1) Denoising tool. We applied r.denoise tool in Grass GIS, but got some problems with the module installation. We ran it again with a shell in Windows, and received an insufficient memory message.

2) Filters. We ran different types of filters (low-pass, median, mean, etc.), with different windows sizes, and trying to put weight in the direction of the streets (Grass GIS, SAGA, QGIS).

3) Geostatistics. We generated points cloud strictly over streets (tried 1000 and 2000 points), generated a variogram model and then ran an ordinary krigging to fill blocks. Variogram modelling, and ordinary krigging, was done in R, using different packages. We got a linear variogram, so we wouldn't rely on kriging results.

4) Other tools. Installed ALDPAT tool, but couldn't get it to work because the program couldn't read the DEM.

In all cases, results in terms of the drainage network wasn't good, because we couldn't avoid the rectangular stream network; also, TWI still resulted in a blocks pattern.

In particular, with OK interpolated result, we got a point-like pattern DEM that affected the network result. However, the effect of the blocks pattern was diminished.

Also, we took a look at this question and answers...

Filtering out canopies and buildings from DSM to have a bare earth elevation

...which redirected us to Whitebox Geospatial Analysis Tools, but couldn't convert our DEM to LAS format. Also, we were not sure about the effectiveness of Bare-Earth DEM tool for us, because it's designed for removing semi-transparent objects, not blocks incorrectly "uplifted", which is our case.

We still want to generate a high quality DEM to make our hydrographic analysis, but don't know what else can we try.

Answer 1

If you are open to using alternative software to solve your problem, then I can suggest the Remove Off-Terrain Objects tool of the cross-platform open-source GIS Whitebox Geospatial Analysis Tools (of which I am lead developer). I realize that you said in your question that you could not convert your data to LAS format, but the tool takes a raster, not LAS file, as the input. I think you were perhaps mixing this tool up with the Bare-Earth DEM tool, which does take a LAS input. You can import your raster DEM into Whitebox as a GeoTIFF file or an ArcGIS floating point binary raster (.flt) or any number of other common raster formats.

Here is another example of it's ability to remove buildings from a raster DEM:

Importantly, the algorithm is not a filter and therefore will not impact the elevations of grid cells outside of 'off-terrain objects' (buildings). This is critical if you want to use your bare-Earth DEM to calculate the wetness index. (That said, I'd question the usefulness of estimating the wetness index of an urban or dense suburban location. Furthermore, there is no stream network through the dense urban landscape you show in your image...I'm certain most of the streams have been culverted.)

EDIT

Actually, looking at the sample dataset that you uploaded, I don't know if your DEM is suited to use with the Remove Off-terrain Objects tool. I had thought that the image that you uploaded was simply suffering from poor symbology, but I see now that it is in fact terraced (i.e. there are large flat areas steps within the DEM). Take a look at the following profile from your DEM:

The tool relies on accurate measures of slope (which I don't think are possible with your DEM...this should be flagged up if you want to calculate TWI as well) and the preservation of sharp slope boundaries between the ground surface and the buildings. But in your case the DEM is severely smoothed and these sharp edges aren't as apparent (your houses also appear to be about 2-3 m in height, which is a bit odd). Are you able to get your hands on the original unprocessed LiDAR DEM, or better yet, the point cloud data? For your application, I would seriously consider re-interpolating the DEM.

Answer 2

I would have to say that without the original LAS point cloud, you will only be incorporating more inaccuracies into the data through raster manipulation. The DEM provided looks relatively clean for a heavily urbanized 1m resolution DEM. The "uplifted squares" are a result of triangulations across data voids, where the buildings are not included in the final ground class. You may also want to consider that water will not flow across a bare earth DEM in the real world as it does in your model. The buildings will present boundaries to actual hydrographic processes.