1

I have written an application in c++ to read LiDAR files and convert to various rasters using GDAL; this worked very well on small scale tests (47 LAS files) however, when used on a live task the memory usage climbs although no further memory is allocated by the application. It appears that the memory is ending up in "mapped file" memory (using RamMap by sysinternals).

Given that this is a 32bit application on a 64bit system I suspect the next call for memory goes beyond the 4GiB limit from the initial memory allocation and the application is no longer able to continue; debug shows that the memory is unallocated with an address of 0x0000. Going into the read loop the application is approximately half to three quarters of a gigabyte (hard to tell from task manager) after reading 8000 LAS files memory is 9+ GB (7GB of "mapped file" with less than 1MB in the standby column).

During the looping I am using fstream to read the LAS file and GDALDataset->RasterIO with GDALRWFlag::GF_Read and GDALRWFlag::GF_Write to handle raster reading and writing.

Can anyone advise on what is causing the memory to be used by RasterIO and more importantly how to reduce this or free up the memory after I/O.

If anyone has encountered similar situations would I be best to write to blocks emulating rasters (like ESRI BIL format) using fstream and making them into real rasters at the last possible moment or is hanging onto the memory a windows 7 problem?

Some stats that may help: OS : Windows 7 Professional 64bit. RAM : 16GB. PageFile : None. i7 CPU, I/O occurs on RAID 5 volume (but has been tested on single drive with similar results)

Sorry, no code is available.. I might get into trouble for sharing.

I'll try my best to express as pseudo code..

I start by reading all of the LAS files in a directory and then using a min/max 
combine into an overall extent called unimaginatively Xmin,Ymin,Xmax and Ymax then 
create the rasters to cover this extent.

    double GeoTransform[6];
    GeoTransform[0] = Xmin; 
    GeoTransform[1] = CellSize;
    GeoTransform[2] = 0;                    
    GeoTransform[3] = Ymax;             
    GeoTransform[4] = 0;                    
    GeoTransform[5] = -CellSize;

    GDALDataset* MinRasDS; // There's an if statement before initializing this
    MinRasDS = (GDALDataset*) IMGdriver->Create(MinRastName,Cols,Rows,1,GDALDataType::GDT_Float32,NULL);
    MinRasDS->SetGeoTransform(GeoTransform);
    MinRasDS->SetProjection(SR_Text);

Create a memory block big enough to contain enough of the big raster to cover a single LiDAR tile
float* MinRastData; // also an if statement here, the memory is only allocated if min raster is selected
MinRastData   = new float[ArrayElements];

Create a memory block big enough to grab some LAS records
char* RecordBlock = new char[Gulp * SizeOfRecord]; // always created

then loop for each file
for (fCnt = 0;fCnt < FileCount;fCnt++)
{
    establish the extent of this file using the header +/- 50 cells as LiDAR extents are imprecise
    pFile.open(NodeIndex[fCnt]->FullPath,ios::in | ios::binary);
    pFile.read(Fpart,sizeof(LAShdrFirst));
    pFile.seekg(131,ios::beg); // Header spatial starts at 131
    pFile.read(SpatPrt,sizeof(LAShdrSpatial));
    LAShdrFirst* HeadrFirst  = (LAShdrFirst*) Fpart;
    LAShdrSpatial* HeadrSpat = (LAShdrSpatial*) SpatPrt;

    TileXmin = HeadrSpat->MinX - (50 * CellSize);
    TileYmax = HeadrSpat->MaxY + (50 * CellSize);
    TileXmax = HeadrSpat->MaxX + (50 * CellSize);
    TileYmin = HeadrSpat->MinY - (50 * CellSize);

    calculate the offset and the size of this extent to the larger raster and
    then read in the values; there is no guarantee tiles are not overlapping
    BlockCols = (TileXmax - TileXmin) / CellSize;
    BlockRows = (TileYmax - TileYmin) / CellSize;
    ColOff = (TileXmin - Xmin) / CellSize; // how many cells to get here!
    RowOff = (Ymax - TileYmax) / CellSize; // top-down reference

    MinRasDS->RasterIO (GDALRWFlag::GF_Read,ColOff,RowOff,BlockCols,BlockRows,
        MinRastData,BlockCols,BlockRows,GDALDataType::GDT_Float32,1,NULL,0,0,0);

    pFile.seekg(HeadrFirst->Offset_to_Data,ios::beg); // go to the first record

    Read a chunk of las records 
    pFile.read(RecordBlock,Gulp * SizeOfRecord);
    for (pCnt = 0;pCnt < Gulp;pCnt++)
    {
        Type1 = (LASrec1*) &RecordBlock[pCnt*SizeOfRecord];
        modify the values in the array using each point
    }
    MinRasDS->RasterIO (GDALRWFlag::GF_Write,ColOff,RowOff,BlockCols,BlockRows,
        MinRastData,BlockCols,BlockRows,GDALDataType::GDT_Float32,1,NULL,0,0,0);
    MinRasDS->FlushCache();
    pFile.close();
}

delete[] RecordBlock;
delete[] MinRastData; // provided it was created first

I call FlushCache() every time to ensure the data is written and ready for the next read which due to alphanumeric ordering is almost certain to be adjacent and overlapping by at least 100 pixels.

This pseudo code is very simplified, in truth there is up to 7 rasters being created and only the min raster is shown. I have several printf statements within the block to show progress so I can track where the process is up to. It is while the program is reading the LAS files that the memory increases, from 4GB used on the first file up to 9GB used on the 8000th file.

Using SysInternals RamMap I found that the data building up is in the system working set which I could empty periodically with no detriment to running processes. This is not an ideal situation as most of this processing is done overnight.

3
  • Can you share psuedo code? Are you sure it's not your file reading causing issues. Also call GDALDataset::FlushCache() occasionally. Tough to help w/o a little code. Even just an example loop with fread/fstream and GDALRasterIO would help.
    – user10353
    Oct 10, 2013 at 22:39
  • I don't see anything, no allocation in the loop that would cause a leak. Are you freeing MinRasData for each file? Sorry I can't help more.
    – user10353
    Oct 15, 2013 at 14:59
  • No, MinRasData is allocated before the loop to the largest pixel size possible and then freed after the loop. There are no new or malloc in the loop - I've been through it a few times now and I'm quite sure that there are no memory allocations within the loop (on the user code anyway). Oct 15, 2013 at 21:41

1 Answer 1

1

The problem does not come from RasterIO, when this same executable is run on Windows XP it does not overuse memory.

The revelation by RamMap of where the memory was being consumed lead me to search along that path.

Windows 7 is an excellent product, but it is built to be generic with the same memory management for home use as for processing large amounts of data. Windows 7 (and I assume 8) keeps old pages in memory just in case they are needed again and does not free them up until all the installed RAM is full, and then only grudgingly; to speed up disc I/O Windows will not necessarily commit to disc when flush() is called but rather put it in a queue to be written to the disc when the device is idle/available, this allows the program to continue quicker and the lag us usually only milliseconds, which is fine for a spreadsheet/word doc but not so good in the case where the program is reading from and writing to the device more data than the available RAM/page.

To get around this I found and installed Windows Dynamic Cache which starts to dump old pages when memory (RAM) usage gets around 90% allowing malloc() requests to obtain the required memory, though not more than 10% of installed RAM. This also allows other applications to start, releasing more old pages as the application consumes new memory.

The other way to get around the cache-stay-resident phenomenon is to open the file sans cache using FILE_FLAG_NO_BUFFERING in CreateFile, but because I'm using the GDAL library as compiled, which in turn relies on libraries already complied, I do not have access to that level of IO.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.