10

Is it possible to read raw data into a geopandas GeoDataFrame, a la a pandas DataFrame?

For example, the following works:

import pandas as pd
import requests
data = requests.get("https://data.cityofnewyork.us/api/geospatial/arq3-7z49?method=export&format=GeoJSON")
pd.read_json(io.BytesIO(r.content))

The following does not:

import geopandas as gpd
import requests
data = requests.get("https://data.cityofnewyork.us/api/geospatial/arq3-7z49?method=export&format=GeoJSON")
gpd.read_file(io.BytesIO(r.content))

In other words, is it possible to read geospatial data that's in memory without saving that data to disk first?

13

You can pass the json directly to the GeoDataFrame constructor:

import geopandas as gpd
import requests
data = requests.get("https://data.cityofnewyork.us/api/geospatial/arq3-7z49?method=export&format=GeoJSON")
gdf = gpd.GeoDataFrame(data.json())
gdf.head()

Outputs:

                                            features               type
0  {'type': 'Feature', 'geometry': {'type': 'Poin...  FeatureCollection
1  {'type': 'Feature', 'geometry': {'type': 'Poin...  FeatureCollection
2  {'type': 'Feature', 'geometry': {'type': 'Poin...  FeatureCollection
3  {'type': 'Feature', 'geometry': {'type': 'Poin...  FeatureCollection
4  {'type': 'Feature', 'geometry': {'type': 'Poin...  FeatureCollection

For supported single-file formats or zipped shapefiles, you can use fiona.BytesCollection and GeoDataFrame.from_features:

import requests
import fiona
import geopandas as gpd

url = 'http://www.geopackage.org/data/gdal_sample.gpkg'
request = requests.get(url)
b = bytes(request.content)
with fiona.BytesCollection(b) as f:
    crs = f.crs
    gdf = gpd.GeoDataFrame.from_features(f, crs=crs)
    print(gdf.head())
and for zipped shapefiles (supported as of fiona 1.7.2)
url = 'https://www2.census.gov/geo/tiger/TIGER2010/STATE/2010/tl_2010_31_state10.zip'
request = requests.get(url)
b = bytes(request.content)
with fiona.BytesCollection(b) as f:
    crs = f.crs
    gdf = gpd.GeoDataFrame.from_features(f, crs=crs)
    print(gdf.head())

You can find out what formats Fiona supports using something like:

import fiona
for name, access in fiona.supported_drivers.items():
    print('{}: {}'.format(name, access))

And a hacky workaround for reading in-memory zipped data in fiona 1.7.1 or earlier:

import requests
import uuid
import fiona
import geopandas as gpd
from osgeo import gdal

request = requests.get('https://github.com/OSGeo/gdal/blob/trunk/autotest/ogr/data/poly.zip?raw=true')
vsiz = '/vsimem/{}.zip'.format(uuid.uuid4().hex) #gdal/ogr requires a .zip extension

gdal.FileFromMemBuffer(vsiz,bytes(request.content))
with fiona.Collection(vsiz, vsi='zip', layer ='poly') as f:
    gdf = gpd.GeoDataFrame.from_features(f, crs=f.crs)
    print(gdf.head())
  • This works for GeoJSON, which answers the question. But this wouldn't work for other geospatial file formats, things like shapefiles or KML or KMZ. Do you know of a workaround for those cases? – Aleksey Bilogur Jan 24 '17 at 1:07
  • A little clarification is in order. GeoPandas and Fiona do support shapefiles and KML, but they can't fully support one off APIs like the City of New York's. Also, BytesCollection totally works, but is probably going to be removed in a future version in favor of one of the options in github.com/Toblerity/Fiona/issues/409. – sgillies Jan 24 '17 at 8:59
  • Thanks. @sgillies should this be opened as a feature request on geopandas, or would it be better to wait for the changes you mention here? – Aleksey Bilogur Jan 25 '17 at 0:35
  • @sgillies you state Fiona supports KML in your comment above, but DriverError: unsupported driver: 'KML' is raised when attempting to open KML as it's not in the supported_drivers dict (using Fiona 1.7.1) and I noticed a couple of issues re. lack of KML support (#23 & #97). Does Fiona support KML? – user2856 Jan 26 '17 at 22:23
2

Yes, it is possible now with Fiona (see https://github.com/Toblerity/Fiona/issues/409). I'm not sure if this feature is exposed yet in Geopandas.

1

Since fiona.BytesCollection doesn't seem to work for TopoJSON here an solution that works for all without the need of gdal:

import fiona
import geopandas as gpd
import requests

# parse the topojson file into memory
request = requests.get('https://vega.github.io/vega-datasets/data/us-10m.json')
visz = fiona.ogrext.buffer_to_virtual_file(bytes(request.content))

# read the features from a fiona collection into a GeoDataFrame
with fiona.Collection(visz, driver='TopoJSON') as f:
    gdf = gpd.GeoDataFrame.from_features(f, crs=f.crs)
  • With geopandas==0.4.0, Fiona==1.8.4 and Python 3, I get DriverError: unsupported driver: 'TopoJSON'. – edesz Jan 20 at 1:22
  • You are right. It was working till at least version 1.7.13 of Fiona – Mattijn Jan 20 at 9:00
  • It is unfortunate that this doesn't work. I was trying to follow your example on GitHub for Altair choropleth plots but that too throws the exact same error at the line gdf = gpd.read_file(counties, driver='TopoJSON'). I thought that using with fiona.Collection... might work but sadly it doesn't. – edesz Jan 20 at 17:20
  • @edesz this was a bug and will be fixed in Fiona 1.8.5, see: github.com/Toblerity/Fiona/issues/721 – Mattijn Feb 25 at 21:25
0

The easiest way is inputting the GeoJSON URL directly into gpd.read(). I'd tried extracting a shapefile from a zip before this using BytesIO & zipfile and had issues with gpd (specifically Fiona) accepting file-like objects.

import geopandas as gpd
import David.SQL_pull_by_placename as sql
import os

os.environ['PROJ_LIB'] = r'C:\Users\littlexsparkee\Anaconda3\Library\share\proj'

geojson_url = f'https://github.com/loganpowell/census-geojson/blob/master/GeoJSON/500k/2018/{sql.state}/block-group.json?raw=true'
census_tracts_gdf = gpd.read_file(geojson_url)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.