Using Microsoft Footprint GeoJSON for spatial analysis with GDAL?

Question

I am trying to use the Microsoft Footprint data (available as GeoJSON) for spatial analysis to roughly identify buildings over a certain size within a specific range of major railroads and highways. I am confident that I know how to perform the analysis, but I can not find a way to use a GeoJSON file as large as these (e.g. Alabama approx. 500 mb of txt) for spatial analysis.

I can very slowly load the GeoJSON into QGIS, but I cannot export as a shapefile or run any search-by-locations without the whole system crashing.

I have tried to use ArcMap's JSON to Feature, but it will not read a GeoJSON of that size.

I have also tried to create individual shapefiles from each feature within the GeoJSON to avoid having a shapefile over the size limit. My intention was to then include these all in a GDB for analysis, but this process too constantly crashed my Python instance and even unmounted a disk.

from osgeo import ogr
import fiona
import json

text = open('Alabama.txt', 'r')
text = text.read()
search = re.findall('{"type":"Polygon","coordinates"[^}]+', text)

i = 0

for line in search:
    i += 1
    poly = line + '}'
    polyogr = ogr.CreateGeometryFromJson(poly)
    schema = {'geometry': 'Polygon','properties': {'fld_a': 'str:50'}}
    with fiona.open('%s.shp' % i, 'w', 'ESRI Shapefile', schema) as layer:
        layer.write({'geometry': json.loads(poly), 'properties': {'fld_a': 'test'}})  

Can I use GeoJSON files this large for analyst work in GDAL?

Answer 1

GeoJSON is a very verbose text format and therefore results in huge files and memory overhead, so converting huge geojson files to something else is certainly a good approach. You don't state the specification of your machine but since QGIS can't load it effectively and you are getting a lot of crashes, I would use ogr2ogr from the command line for the conversion and thereby remove the overhead of running QGIS itself (I would also make sure I have nothing else runing on my machine too). As you have QGIS, you won't need to install ogr2ogr as you should have it already (you may need to ensure your machine's environment variables are suitably set and you have your paths right to ensure it works).

In the ogr2ogr documentation, note the options you have for clipping the data and for controlling performance (especially the -gt option). ogr2ogr can output to a huge number of different formats and for something this big I might go down the route of a PostGIS database and then you can do some SQL queries to fan out the data (or not) at your leisure and with less risk of crashing. Recent versions of opgr2ogr have had improvements to reduce their memory overhead, so this is where I'd start.