3

This question already has an answer here:

Are there any publicly and free (not necessarily open-source) applications that can remove duplicate points from LiDAR clouds?

I would like something on the lines of lasduplicate -unique_xyz, which seems to work fine, but LasTools are no longer free and have certain input limits above which the output is tainted by noise, missing information and it also seems to be incomplete.

Also, I've tried SAGA GIS's "remove duplicate points", but this option is almost impossible to work with for large sets (more than a couple of million and SAGA GIS hangs).

LATER EDIT: a Python script able to filter out such an input is provided in my answer to this question. You should force a garbage collect from time to time if you run into memory problems.

marked as duplicate by Andre Silva, xunilk, BERA, aldo_tapia, Dan C Jan 4 '18 at 19:50

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • Yep, it's having problems with las with too many points. But it works! Haha To install laspy for 64bit Python, just paste the contents of the zip/tar/rar file into \Lib\site-packages. Then open IDLE and import laspy. – Tox Jul 24 '14 at 7:51
  • Could you add some information explaining how this answers the question? – whuber Jul 24 '14 at 13:24
  • On a 32bit version, the script will run out of memory if the files are too big (try maybe 30million points). – Tox Jul 25 '14 at 2:57
  • @AndreSilva Isn't How to delete duplicate LiDAR points a duplicate of this older question? – nmtoken Jan 4 '18 at 18:20
  • @nmtoken, it can be; it is a matter of choice. Not always the older question should be the canonical one (meta.stackexchange.com/questions/10841/…), but the one with better collection of answers. In this case, I judged this Q has only one answer which shows the 'how to', the other has an answer more plastic than this one; i..e., it is easier to use. Also, the other post is more of type 'how to do', this one is 'which software do'; and I always prefer the former type. continues... – Andre Silva Jan 4 '18 at 18:37
5

If you are open to scripting, I've used several Python libraries to process large (>30 million points) LAS point clouds. The best one I've found is laspy. It easily reads LAS files into a numpy array, and from there its as simple as filtering and writing to a new file.

  • 1
    @AndreSilva, I'm almost there regarding the implementation. Once I have it (ETA 1 day), I'll post an answer. I'll accept this as the answer, though, because of the initial suggestion. – teodron Mar 10 '14 at 19:26
  • Alright, I concocted a Python script to do that processing. I have never programmed anything in Python and it was very difficult to even install its libraries. I couldn't install it manually, so I resorted to a few 32 bit installers for all libraries. I'd be grateful if anyone could point out a quick and efficient tutorial on how to do install laspy for a 64bit Python version (I guess it means using an egg file or something). On a 32bit version, the script runs out of memory for decently big files (10M points). Any help will be much appreciated. – teodron Mar 11 '14 at 16:32
4

Based on Barbarossa's answer, I managed to put together this rather inefficient but sometimes functional Python script. It uses the laspy library. It is memory intensive and may fail when ran on 32bit versions of Python. I actually ran out of memory when processing a file of 10M points.

import numpy as np
from laspy.file import File

inFile = File("input.las", mode="r")

#artificial indices - serve to recover the whole point information from inFile.points
artificialIndices = np.arange(len(inFile.x), dtype = int)

# create an artificial numPy array using x y z classification and index
coords = np.vstack((inFile.x, inFile.y, inFile.z, inFile.classification, artificialIndices)).transpose()

# first, sort the 2D NumPy array row-wise so dups will be contiguous
# and rows are preserved
a, b, c, d, e = coords.T    # create the keys for to pass to lexsort
ndx = np.lexsort((a, b, c))

# replace the array inplace with the ordered sequence
coords = coords[ndx,]

# free up some memory (x86 really needs it)
del ndx
del a
del b
del c
del d
del e

# how many input points
numRows = coords.shape[0]
# fake indices pointing towards the initial inFile.points
indices = np.zeros(numRows, dtype = int)
duplicates = 0;
singles = 0;
idx = 0
index = 0;
while (idx < numRows):

    jdx = idx + 1;
    singles = singles + 1;

    while (jdx < numRows and (coords[idx, 0:3] == coords[jdx, 0:3] ).all() ):
        duplicates = duplicates + 1
        once = True
        if once:
            if (jdx < 1000):
                print int(coords[idx][4]), " -- ", coords[idx][0], coords[idx][1], coords[idx][2], coords[idx][3]
            once = False
        jdx = jdx + 1

    indices[index] = int(coords[idx][4])
    index = index + 1

    idx = jdx

print "duplicate count = ", duplicates, "single count =", singles


del coords # do not need it anymore

# slice the input points and keep only the ones stored in the indices array
points_kept = inFile.points[indices[0:index]]



print("Writing output files...")
outFile1 = File("output.las", mode = "w", header = inFile.header)
outFile1.points = points_kept
outFile1.close()

print("Closing input file...")
inFile.close()

Not the answer you're looking for? Browse other questions tagged or ask your own question.