4

I have recently downloaded a building footprint GeoJSON file from Microsoft Bing Maps. The file I downloaded was for the entirety of Texas, and is of approximately 2.2 GB file size. They do not provide the data in any smaller sections for the state of Texas.

My issue is that I am unable to convert this GeoJSON into any usable format, such as a shapefile. I have tried to use https://mapshaper.org/ but my file size is too large and the website crashes. Similarly, I have tried to open the GeoJSON in QGIS, but that program crashes as well. I know ArcGIS has the Data Interoperability Toolset, but that requires a special license.

Does anyone have any other techniques to convert a large GeoJSON file?

  • 2
    You know that these are already in Shapefile Format here? wiki.openstreetmap.org/wiki/Microsoft_Building_Footprint_Data Example > Texas direct download 1drv.ms/u/s!AqWv0F0N63JkgSxCOnhjhJGlsVHx – Mapperz Oct 10 '18 at 16:39
  • 3
    personally I'd say don't use shapefiles, and not in favor of GeoJSON (or at least not in an OGC aware environment). note that e.g. due to interoperability concerns, 2GB should be the working size maximum (not the technical limit, though). QGIS comes shipped with the core GDAL/OGR (terminal) functions; if you like to give it a try, ogr2ogr can do some magic to your file, including splitting it into multiple files and format conversion. – ThingumaBob Oct 10 '18 at 16:56
  • @ThingumaBob They stated in the question that QGIS crashes when trying to load the geojson file. Is it possible to use ogr2ogr to split the geojson without first loading it into QGIS? If so, that would be helpful if you posted it as an answer. – csk Oct 10 '18 at 18:49
  • 3
    @csk oh it is, ogr2ogr is run from the terminal and doesn't need QGIS...the other way around maybe. but while writing the answer I tried my script on a generated 2.5GB file (~25M rows) and was struck by failure: the driver doesn't read the file in chunks properly, even with -gt option set, and ogr2ogr killed itself due to memory overhead. I did convert ~1.9GB files on a much larger machine than my laptop, but this might not be practical then. this however, was gold. – ThingumaBob Oct 10 '18 at 20:42
  • 1
    Thank you all very much for the help! @ThingumaBob, the link to the python script was perfect, I was able to split the geojson into 4 parts and then separately convert them all to shapefiles in QGIS. – M. Gilliam Oct 12 '18 at 20:57
3

Shapefile have a size limit of 2GB so trying to convert your 2.2GB geojson probably result in a too big shapefile.

You should try to load it in a database that is capable of handling that amount of data (a simple spatialite database for exemple could be created and managed from qgis and has no size limit)

Also for large file you may run to hardware limitation....

  • 2
    A lot depends on the attributes, but I wouldn't expect a 2.2Gb GeoJSON to exceed 1Gb in shapefile. – Vince Oct 10 '18 at 17:44
  • 3
    Those building footprints do not contain any attributes at all. Conversion with something like ogr2ogr -f "ESRI Shapefile" texas.shp texas.geojson should work. – user30184 Oct 10 '18 at 18:56
2

https://github.com/woodb/geojsplit works for me.

node --max-old-space-size=20480 /usr/lib/node_modules/geojsplit/bin/geojsplit -a 1 -l 3000000 -v -o ~/data ~/data/California.geojson

I have divided California.geojson into four files with max 3000000 features per each.

By default, node has only 512 Mb memory limit, so --max-old-space-size=20480 increase it to 20Gb

0

INSTALL NODE FIRST -- https://www.guru99.com/download-install-node-js.html

OPEN powershell install $geojsplit PS c:\users\rakesh>$geojsplit

and now go to your geojson file location and run this script

PS D:\Florida>node --max-old-space-size=500 C:\Users\rakesh\AppData\Roaming\npm\node_modules\geojsplit\bin\geojsplit -a 1 -l 30000 -v -o k D:/florida/florida.geojson

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.