Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

As a followup to my previous question (Converting SimpleGeo Places GeoJSON file to shapefile), I am now struggling with what I perceive might be an ogr2ogr "memory" limitation, but I cannot be sure. I have successfully converted the SimpleGeo dump files into proper geojson files (verified that are directly readable in QGIS).

Now I want to convert this data to SHP or other formats (SQLite, PostGIS). I am running the following command with success on smaller files (where "places_fr.gejson" has a file size of ~118MB):

ogr2ogr -f "ESRI Shapefile" places_fr.shp places_fr.geojson

When I attempt to run this same exact command with a geojson file that is 340MB, ogr2ogr crashes in one of two ways:

  1. "lh_table_new: calloc failed" (first attempt)
  2. Windows throws an "ogr2ogr.exe has stopped working. Please close the program." (subsequent attempts)

I am running the ogr command via FWTools in a Windows 7 environment dual core with 8GB of installed RAM (4GB available, 12GB virtual memory available). Is there a memory limit to ogr2ogr converting these datasets? I have also tried converting into SQLite with the same results, so I don't think it is an SHP conversion issue. Thanks.

share|improve this question
add comment

3 Answers 3

up vote 6 down vote accepted

Forewarning: given the amout of memory you are reporting on your system I suppose you are using a 64 bit build of Windows. If not this solution does not apply.

The memory limit for all 32 bit applications on Windows (regardless of the Windows version, so it is true also for 64 bit Windows) is 2GB. You might be hitting that limit, to confirm run ogr2ogr and check memory usage with task manager.

If the problem is indeed due to a memory issue you can do two things:

  1. build yourself a 64 bit version of GDAL (I could not find readily avail binaries)

  2. patch ogr2ogr.exe with this tool: http://ntcore.com/4gb_patch.php. After patching (make a backup copy of course) the binary should be available to address a grandtotal 4GB

share|improve this answer
1  
I watched the resource monitor, and as soon as the memory reached 2GB, the ogr2ogr process crashed. It was definitely a memory limitation of 2GB within ogr2ogr/fwtools, likely because it was compiled as a 32-bit app. I patched the ogr2ogr.exe with the tool you suggested and it worked like a charm. Of course, there still is a limit where it cannot be used (like with the 7GB U.S. places files), but at least there is an explanation for why ogr2ogr is failing. Thanks! –  RyanDalton Oct 29 '11 at 16:46
add comment

If there is a problem, it would be most likely specific to the OGR geojson driver and not OGR itself. Try using a newer binary (like the one from OSGeo4W) or even better, a 64 bit version of it. Sorry, unless you are compiling it, I do not know where you can get a 64bit binary for Windows - the Ubuntu Linux and Mac KyngChaos binaries will be 64 bit though.

share|improve this answer
add comment

May be split the file to many files and import with a batch the files.

http://www.clubic.com/telecharger-fiche184550-split-files.html or http://www.hjsplit.org/

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.