ogr2ogr fails to convert GeoJson file > 200MB

Question

As a followup to my previous question (Converting SimpleGeo Places GeoJSON file to shapefile), I am now struggling with what I perceive might be an ogr2ogr "memory" limitation, but I cannot be sure. I have successfully converted the SimpleGeo dump files into proper geojson files (verified that are directly readable in QGIS).

Now I want to convert this data to SHP or other formats (SQLite, PostGIS). I am running the following command with success on smaller files (where "places_fr.gejson" has a file size of ~118MB):

ogr2ogr -f "ESRI Shapefile" places_fr.shp places_fr.geojson

When I attempt to run this same exact command with a geojson file that is 340MB, ogr2ogr crashes in one of two ways:

1. "lh_table_new: calloc failed" (first attempt)
2. Windows throws an "ogr2ogr.exe has stopped working. Please close the program." (subsequent attempts)

I am running the ogr command via FWTools in a Windows 7 environment dual core with 8GB of installed RAM (4GB available, 12GB virtual memory available). Is there a memory limit to ogr2ogr converting these datasets? I have also tried converting into SQLite with the same results, so I don't think it is an SHP conversion issue. Thanks.

Answer 1

Forewarning: given the amout of memory you are reporting on your system I suppose you are using a 64 bit build of Windows. If not this solution does not apply.

The memory limit for all 32 bit applications on Windows (regardless of the Windows version, so it is true also for 64 bit Windows) is 2GB. You might be hitting that limit, to confirm run ogr2ogr and check memory usage with task manager.

If the problem is indeed due to a memory issue you can do two things:

1. build yourself a 64 bit version of GDAL (I could not find readily avail binaries)

2. patch ogr2ogr.exe with this tool: http://ntcore.com/4gb_patch.php. After patching (make a backup copy of course) the binary should be available to address a grandtotal 4GB

Answer 2

May be split the file to many files and import with a batch the files.

http://www.clubic.com/telecharger-fiche184550-split-files.html or http://www.hjsplit.org/

Answer 3

If there is a problem, it would be most likely specific to the OGR geojson driver and not OGR itself. Try using a newer binary (like the one from OSGeo4W) or even better, a 64 bit version of it. Sorry, unless you are compiling it, I do not know where you can get a 64bit binary for Windows - the Ubuntu Linux and Mac KyngChaos binaries will be 64 bit though.