I am trying to use the Microsoft Footprint data (available as GeoJSON) for spatial analysis to roughly identify buildings over a certain size within a specific range of major railroads and highways. I am confident that I know how to perform the analysis, but I can not find a way to use a GeoJSON file as large as these (e.g. Alabama approx. 500 mb of txt) for spatial analysis.

I can very slowly load the GeoJSON into QGIS, but I cannot export as a shapefile or run any search-by-locations without the whole system crashing.

I have tried to use ArcMap's JSON to Feature, but it will not read a GeoJSON of that size.

I have also tried to create individual shapefiles from each feature within the GeoJSON to avoid having a shapefile over the size limit. My intention was to then include these all in a GDB for analysis, but this process too constantly crashed my Python instance and even unmounted a disk.

from osgeo import ogr
import fiona
import json

text = open('Alabama.txt', 'r')
text = text.read()
search = re.findall('{"type":"Polygon","coordinates"[^}]+', text)

i = 0

for line in search:
    i += 1
    poly = line + '}'
    polyogr = ogr.CreateGeometryFromJson(poly)
    schema = {'geometry': 'Polygon','properties': {'fld_a': 'str:50'}}
    with fiona.open('%s.shp' % i, 'w', 'ESRI Shapefile', schema) as layer:
        layer.write({'geometry': json.loads(poly), 'properties': {'fld_a': 'test'}})  

Can I use GeoJSON files this large for analyst work in GDAL?

  • If you also want to ask about doing this using ArcGIS Desktop and/or QGIS please feel free to do that in separate questions. – PolyGeo Oct 31 '18 at 19:35

GeoJSON is a very verbose text format and therefore results in huge files and memory overhead, so converting huge geojson files to something else is certainly a good approach. You don't state the specification of your machine but since QGIS can't load it effectively and you are getting a lot of crashes, I would use ogr2ogr from the command line for the conversion and thereby remove the overhead of running QGIS itself (I would also make sure I have nothing else runing on my machine too). As you have QGIS, you won't need to install ogr2ogr as you should have it already (you may need to ensure your machine's environment variables are suitably set and you have your paths right to ensure it works).

In the ogr2ogr documentation, note the options you have for clipping the data and for controlling performance (especially the -gt option). ogr2ogr can output to a huge number of different formats and for something this big I might go down the route of a PostGIS database and then you can do some SQL queries to fan out the data (or not) at your leisure and with less risk of crashing. Recent versions of opgr2ogr have had improvements to reduce their memory overhead, so this is where I'd start.

| improve this answer | |
  • 2
    @Onimonipea if you use ogr2ogr, which I recommend you do, I suggest you use GDAL 2.3 or later as previous versions read the entire file into memory (trac.osgeo.org/gdal/ticket/6540). – user2856 Nov 1 '18 at 5:25
  • This worked perfectly. Thank you. I hadn't been able to get ogr2ogr to work previously, but this answered my question. – Onimonipea Nov 1 '18 at 15:51
  • Quick update. This method has worked so far, but I have just tried it on the California.geoson and received this. ogr2ogr -nlt POLYGON -skipfailures California.shp California.geojson Killed: 9 Any advice? – Onimonipea Nov 2 '18 at 17:45
  • You may have simply reached the limit of the antedeluvian shapefile format. Loading that volume of data would be better in a transactional database like PostGIS and then you can use the -gt switch. – MappaGnosis Nov 3 '18 at 9:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.