8

I have recently downloaded a building footprint GeoJSON file from Microsoft Bing Maps. The file I downloaded was for the entirety of Texas, and is of approximately 2.2 GB file size. They do not provide the data in any smaller sections for the state of Texas.

My issue is that I am unable to convert this GeoJSON into any usable format, such as a shapefile. I have tried to use https://mapshaper.org/ but my file size is too large and the website crashes. Similarly, I have tried to open the GeoJSON in QGIS, but that program crashes as well. I know ArcGIS has the Data Interoperability Toolset, but that requires a special license.

Does anyone have any other techniques to convert a large GeoJSON file?

Improve this question
6
  • 3
    You know that these are already in Shapefile Format here? wiki.openstreetmap.org/wiki/Microsoft_Building_Footprint_Data Example > Texas direct download 1drv.ms/u/s!AqWv0F0N63JkgSxCOnhjhJGlsVHx
    – Mapperz ♦
    Commented Oct 10, 2018 at 16:39
  • 3
    personally I'd say don't use shapefiles, and not in favor of GeoJSON (or at least not in an OGC aware environment). note that e.g. due to interoperability concerns, 2GB should be the working size maximum (not the technical limit, though). QGIS comes shipped with the core GDAL/OGR (terminal) functions; if you like to give it a try, ogr2ogr can do some magic to your file, including splitting it into multiple files and format conversion.
    – geozelot
    Commented Oct 10, 2018 at 16:56
  • @ThingumaBob They stated in the question that QGIS crashes when trying to load the geojson file. Is it possible to use ogr2ogr to split the geojson without first loading it into QGIS? If so, that would be helpful if you posted it as an answer.
    – csk
    Commented Oct 10, 2018 at 18:49
  • 3
    @csk oh it is, ogr2ogr is run from the terminal and doesn't need QGIS...the other way around maybe. but while writing the answer I tried my script on a generated 2.5GB file (~25M rows) and was struck by failure: the driver doesn't read the file in chunks properly, even with -gt option set, and ogr2ogr killed itself due to memory overhead. I did convert ~1.9GB files on a much larger machine than my laptop, but this might not be practical then. this however, was gold.
    – geozelot
    Commented Oct 10, 2018 at 20:42
  • 1
    Thank you all very much for the help! @ThingumaBob, the link to the python script was perfect, I was able to split the geojson into 4 parts and then separately convert them all to shapefiles in QGIS.
    – M. Gilliam
    Commented Oct 12, 2018 at 20:57

3 Answers 3

Reset to default
5

Shapefile have a size limit of 2GB so trying to convert your 2.2GB geojson probably result in a too big shapefile.

You should try to load it in a database that is capable of handling that amount of data (a simple spatialite database for exemple could be created and managed from qgis and has no size limit)

Also for large file you may run to hardware limitation....

Improve this answer
2
  • 4
    A lot depends on the attributes, but I wouldn't expect a 2.2Gb GeoJSON to exceed 1Gb in shapefile.
    – Vince
    Commented Oct 10, 2018 at 17:44
  • 4
    Those building footprints do not contain any attributes at all. Conversion with something like ogr2ogr -f "ESRI Shapefile" texas.shp texas.geojson should work.
    – user30184
    Commented Oct 10, 2018 at 18:56
3

https://github.com/woodb/geojsplit works for me.

node --max-old-space-size=20480 /usr/lib/node_modules/geojsplit/bin/geojsplit -a 1 -l 3000000 -v -o ~/data ~/data/California.geojson

I have divided California.geojson into four files with max 3000000 features per each.

By default, node has only 512 Mb memory limit, so --max-old-space-size=20480 increase it to 20Gb

Improve this answer
0

INSTALL NODE FIRST -- https://www.guru99.com/download-install-node-js.html

OPEN powershell install $geojsplit PS c:\users\rakesh>$geojsplit

and now go to your geojson file location and run this script

PS D:\Florida>node --max-old-space-size=500 C:\Users\rakesh\AppData\Roaming\npm\node_modules\geojsplit\bin\geojsplit -a 1 -l 30000 -v -o k D:/florida/florida.geojson

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.