writeLAS writes corrupted file: 'end-of-file' after 10468 of 10474 points

Question

I am seeing strange write behavior with lidR. The outputs from lidR are compressed to fall on a single line. I vaguely recall discussing this with JR before, but cannot recall a solution.

#ok
las0=lidR::readLAS("c:\\temp\\tile1.las")
lidR::plot(las0)

#not ok
lidR::writeLAS(las0,"c:\\temp\\tile_lidR.las")
las1=lidR::readLAS("c:\\temp\\tile_lidR.las")
#> ERROR: 'end-of-file' after 10468 of 10474 points
#> Warning messages:
#> 1: There are 113 points flagged 'withheld'. 
#> 2: There are 113 points flagged 'synthetic'.
lidR::plot(las1)

And here are the summaries if I look at the file meta info in R:

> summary(las0) 
class        : LAS (LASF v1.3)
point format : 2
memory       : 872.4 Kb 
extent       :1007414, 1007736, 823583.8, 823586.9 (xmin, xmax, ymin, ymax)
coord. ref.  : NA 
area         : 500.825 units²
points       : 10.5 thoushand points
density      : 20.91 points/units²
names        : X Y Z Intensity ReturnNumber NumberOfReturns ScanDirectionFlag EdgeOfFlightline Classification Synthetic_flag Keypoint_flag Withheld_flag ScanAngleRank UserData PointSourceID R G B 
File signature:           LASF 
File source ID:           0 
Global encoding:
 - GPS Time Type: GPS Week Time 
 - Synthetic Return Numbers: no 
 - Well Know Text: CRS is GeoTIFF 
 - Aggregate Model: false 
Project ID - GUID:        00000000-0000-0000-0000-000000000000 
Version:                  1.3
System identifier:        Trimble 
Generating software:      Inpho 
File creation d/y:        254/2015
header size:              375 
Offset to point data:     375 
Num. var. length record:  0 
Point data format:        2 
Point data record length: 26 
Num. of point records:    10474 
Num. of points by return: 10474 0 0 0 0 
Scale factor X Y Z:       0.001 0.001 0.001 
Offset X Y Z:             1007436 823586 775 
min X Y Z:                1007414 823583.8 722.348 
max X Y Z:                1007736 823586.9 821.588 
Variable length records:  void

> summary(las1)
class        : LAS (LASF v1.3)
point format : 2
memory       : 871.9 Kb 
extent       :1007502, 1007829, 823586, 824962.3 (xmin, xmax, ymin, ymax)
coord. ref.  : NA 
area         : 180387.3 units²
points       : 10.5 thoushand points
density      : 0.06 points/units²
names        : X Y Z Intensity ReturnNumber NumberOfReturns ScanDirectionFlag EdgeOfFlightline Classification Synthetic_flag Keypoint_flag Withheld_flag ScanAngleRank UserData PointSourceID R G B 
File signature:           LASF 
File source ID:           0 
Global encoding:
 - GPS Time Type: GPS Week Time 
 - Synthetic Return Numbers: no 
 - Well Know Text: CRS is GeoTIFF 
 - Aggregate Model: false 
Project ID - GUID:        00000000-0000-0000-0000-000000000000 
Version:                  1.3
System identifier:         
Generating software:      rlas R package 
File creation d/y:        0/2015
header size:              375 
Offset to point data:     375 
Num. var. length record:  0 
Point data format:        2 
Point data record length: 26 
Num. of point records:    10468 
Num. of points by return: 1209 1185 1327 1223 1385 
Scale factor X Y Z:       0.001 0.001 0.001 
Offset X Y Z:             1007436 823586 775 
min X Y Z:                1007502 823586 -2146709 
max X Y Z:                1007829 824962.3 2148193 
Variable length records:  void

The las file can be found here

---

@JR reminded me that the lastools documentation also addresses this case (even the number of bytes) explicitly. I just navigated to the correct directory and overwrote all of the header fields using the command below for the lasinfo tool. I decided to just hardcode changes to the las file headers so that I don't forget and run into this with the same files again.

Martin's code below is updated to use wildcard:

lasinfo -i *.las -set_header_size 235

His code also came with the warning:

CAREFUL! sets the header size field of the LAS header to 235 without checking whether this will corrupt the file.

Answer 1

This question has been discussed here. For an unknown reason the payload of the file is offseted to 375 bytes instead of 235 for a LAS 1.3.

las0@header@PHB$`Header Size`
#> 375 

In theory it could be written properly but in practice rlas generates a corrupted file. You must manually fix the header.

las0@header@PHB$`Header Size` <- 235

Answer 2

It's perfectly valid (although not usual) for the LAS header to contain additional bytes. It seems Trimble always writes 375, no matter if it's LAS 1.2 (227 bytes), LAS 1.3 (235 bytes), or LAS 1.4 (375 bytes). One advantage of this is that the LAS file could be upgraded to LAS 1.4 in place (assuming the point type is kept).

However, those additional 140 bytes in the LAS header (likely just a block of zeros) need to be read and written before reading the block of (optional) Variable Length Records and the subsequent block of LAS point records. Note the "the header contains 140 user-defined bytes" in the lasinfo output.

The file works fine with LAStools version 190623 (which are based on LASlib and LASzip).

C:\LAStools\bin>laszip -i tile1.las

C:\LAStools\bin>lasdiff -i tile1.las
checking 'tile1.las' against 'tile1.laz'
headers are identical.
raw points are identical.
files are identical. both have 10474 points. took 0.051 secs.

C:\LAStools\bin>lasinfo -i tile1.laz
lasinfo (190623) report for 'tile1.laz'
reporting all LAS header entries:
  file signature:             'LASF'
  file source ID:             0
  global_encoding:            0
  project ID GUID data 1-4:   00000000-0000-0000-0000-000000000000
  version major.minor:        1.3
  system identifier:          'Trimble'
  generating software:        'Inpho'
  file creation day/year:     254/2015
  header size:                375
  offset to point data:       375
  number var. length records: 0
  point data format:          2
  point data record length:   26
  number of point records:    10474
  number of points by return: 0 0 0 0 0
  scale factor x y z:         0.001 0.001 0.001
  offset x y z:               1007436 823586 775
  min x y z:                  1007413.640 823583.785 722.348
  max x y z:                  1007736.165 823586.885 821.588
  start of waveform data packet record: 0
the header contains 140 user-defined bytes
LASzip compression (version 3.4r1 c2 50000): POINT10 2 RGB12 2
reporting minimum and maximum for all LAS point record entries ...
  X              -22360     300165
  Y               -2215        885
  Z              -52652      46588
  intensity           1          5
  return_number       1          1
  number_of_returns   1          1
  edge_of_flight_line 0          0
  scan_direction_flag 0          0
  classification      0          0
  scan_angle_rank     0         21
  user_data           0          0
  point_source_ID     1          1
  Color R 6754 45803
        G 9170 46323
        B 9421 42574
number of first returns:        10474
number of intermediate returns: 0
number of last returns:         10474
number of single returns:       10474
WARNING: for return 1 real number of points by return is 10474 but header entry was not set.
overview over number of returns of given pulse: 10474 0 0 0 0 0 0
histogram of classification of points:
           10474  never classified (0)