Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I am attempting to write many ASPRS LAS point files into an ESRI File Geodatabase using my compilation of GDAL 1.9.2. The FileGDB driver for GDAL/OGR appears to be incredibly slow when writing large files, taking as much as 45 minutes to write just 8 million point records. Write speeds of FileGDB using GDAL on a SATA3 drive are somewhere on the order of 200 kilobytes per second, which is unacceptably slow when I'm trying to convert terrabytes of data.

I noticed in the FileGDB documentation that defining the FGDB_BULK_LOAD macro should improve performance for large datasets, but I noticed no change in performance when I wrote a line into the "nmake.opt" file with the text "FGDB_BULK_LOAD = YES" immediately after the FGDB_LIB line.

Admittedly, a FileGDB is not the ideal way to store billions of point data records, but that's a gripe for another time. Have I correctly used the FGDB_BULK_LOAD feature? Is that supposed to be in my source code, not the GDAL build?

Thanks.

UPDATE: Proper Usage: (Answered in chat)

The FGDB_BULK_LOAD setting is properly stored as an environment variable for the GDAL/OGR process. This is set on the command line during the ogr exe call as shown by Ragi. Using the GDAL functions, it can be set programmatically for the entire process with

 CPLSetConfigOption("FGDB_BULK_LOAD", "YES");

or just for the current thread using

 CPLSetThreadLocalConfigOption("FGDB_BULK_LOAD", "YES");

FGDB_BULK_LOAD must be set prior to calling FGdbDataSource::CreateLayer(). It was unclear whether OGRCleanupAll() unset this variable, but it is safe to call multiple times to be sure.

Using that option boosted performance to be around 5.5x faster for writing millions to tens of millions of points.

share|improve this question
add comment

1 Answer 1

up vote 6 down vote accepted

FGDB_BULK_LOAD is not a compilation setting, it is a configuration option for the command line tools (can also be done programmatically).

ogr2ogr --config FGDB_BULK_LOAD YES -f "FileGDB" MyFileGDB.gdb myKML.kml

Would create a filegdb and load the KML vector data to it. Let me know if your performance still sucks. By the way, what platform are you on?

Update:

Just to summarize what was discussed in chat and IRC:

For programmatic access:

 OGRDataSource* datasource = driver->Open("c:/myfilegdb.gdb", TRUE); 
 char* options[] = { "FGDB_BULK_LOAD=YES" , NULL }; 
 OGRLayer newlayer = datasource->CreateLayer("tiny", NULL, wkbPoint25D, options);

should be the equivalent.

share|improve this answer
    
Thank you for your response. I'm on Windows 7x64, coding in C++ on Qt Creator and compiling with MSVC2010. I didn't realize that was a command line option. Is it possible to tap into that performance improvement when coding from the library? –  Phlucious Nov 8 '12 at 19:01
    
General GDAL/OGR gripe: most of the configuration and usage instructions on the driver pages are targetted solely toward commandline usage, whereas my use case requires integration of the GDAL/OGR library into my code. Is there a developer-targetted doc resource that I don't know about? –  Phlucious Nov 8 '12 at 19:02
    
@Phlucious There is the Resources section here: gdal.org/ogr that has plenty of developer documentation. At the end, the best documentation is the source code itself. The entire source code for the FileGDB driver is only 7 files, and it is available here github.com/OSGeo/gdal/tree/trunk/gdal/ogr/ogrsf_frmts/filegdb . In fact, if you want to see how how the flag gets used, you can look here: github.com/OSGeo/gdal/blob/trunk/gdal/ogr/ogrsf_frmts/filegdb/… –  Ragi Yaser Burhum Nov 9 '12 at 18:12
    
The way the setting gets passed around is simply an array of strings. You can find that out by looking here It gets passed as an array of strings github.com/OSGeo/gdal/blob/trunk/gdal/ogr/ogrsf_frmts/filegdb/… –  Ragi Yaser Burhum Nov 9 '12 at 18:13
1  
What you say makes sense. In my application, with feature classes on the order of millions to tens of millions of points, I saw a performance boost of 5.5 to 6.0 times. Pretty impressive! I also learned that you can have multiple threads writing to the same FileGDB as long as they write to different items, which boosts performance even more. –  Phlucious Nov 12 '12 at 21:27
show 5 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.