14

Is it possible to read raw data into a geopandas GeoDataFrame, a la a pandas DataFrame?

For example, the following works:

import pandas as pd
import requests
data = requests.get("https://data.cityofnewyork.us/api/geospatial/arq3-7z49?method=export&format=GeoJSON")
pd.read_json(io.BytesIO(r.content))

The following does not:

import geopandas as gpd
import requests
data = requests.get("https://data.cityofnewyork.us/api/geospatial/arq3-7z49?method=export&format=GeoJSON")
gpd.read_file(io.BytesIO(r.content))

In other words, is it possible to read geospatial data that's in memory without saving that data to disk first?

16

You can pass the json directly to the GeoDataFrame constructor:

import geopandas as gpd
import requests
data = requests.get("https://data.cityofnewyork.us/api/geospatial/arq3-7z49?method=export&format=GeoJSON")
gdf = gpd.GeoDataFrame(data.json())
gdf.head()

Outputs:

                                            features               type
0  {'type': 'Feature', 'geometry': {'type': 'Poin...  FeatureCollection
1  {'type': 'Feature', 'geometry': {'type': 'Poin...  FeatureCollection
2  {'type': 'Feature', 'geometry': {'type': 'Poin...  FeatureCollection
3  {'type': 'Feature', 'geometry': {'type': 'Poin...  FeatureCollection
4  {'type': 'Feature', 'geometry': {'type': 'Poin...  FeatureCollection

For supported single-file formats or zipped shapefiles, you can use fiona.BytesCollection and GeoDataFrame.from_features:

import requests
import fiona
import geopandas as gpd

url = 'http://www.geopackage.org/data/gdal_sample.gpkg'
request = requests.get(url)
b = bytes(request.content)
with fiona.BytesCollection(b) as f:
    crs = f.crs
    gdf = gpd.GeoDataFrame.from_features(f, crs=crs)
    print(gdf.head())
and for zipped shapefiles (supported as of fiona 1.7.2)
url = 'https://www2.census.gov/geo/tiger/TIGER2010/STATE/2010/tl_2010_31_state10.zip'
request = requests.get(url)
b = bytes(request.content)
with fiona.BytesCollection(b) as f:
    crs = f.crs
    gdf = gpd.GeoDataFrame.from_features(f, crs=crs)
    print(gdf.head())

You can find out what formats Fiona supports using something like:

import fiona
for name, access in fiona.supported_drivers.items():
    print('{}: {}'.format(name, access))

And a hacky workaround for reading in-memory zipped data in fiona 1.7.1 or earlier:

import requests
import uuid
import fiona
import geopandas as gpd
from osgeo import gdal

request = requests.get('https://github.com/OSGeo/gdal/blob/trunk/autotest/ogr/data/poly.zip?raw=true')
vsiz = '/vsimem/{}.zip'.format(uuid.uuid4().hex) #gdal/ogr requires a .zip extension

gdal.FileFromMemBuffer(vsiz,bytes(request.content))
with fiona.Collection(vsiz, vsi='zip', layer ='poly') as f:
    gdf = gpd.GeoDataFrame.from_features(f, crs=f.crs)
    print(gdf.head())
|improve this answer|||||
  • This works for GeoJSON, which answers the question. But this wouldn't work for other geospatial file formats, things like shapefiles or KML or KMZ. Do you know of a workaround for those cases? – Aleksey Bilogur Jan 24 '17 at 1:07
  • A little clarification is in order. GeoPandas and Fiona do support shapefiles and KML, but they can't fully support one off APIs like the City of New York's. Also, BytesCollection totally works, but is probably going to be removed in a future version in favor of one of the options in github.com/Toblerity/Fiona/issues/409. – sgillies Jan 24 '17 at 8:59
  • Thanks. @sgillies should this be opened as a feature request on geopandas, or would it be better to wait for the changes you mention here? – Aleksey Bilogur Jan 25 '17 at 0:35
  • @sgillies you state Fiona supports KML in your comment above, but DriverError: unsupported driver: 'KML' is raised when attempting to open KML as it's not in the supported_drivers dict (using Fiona 1.7.1) and I noticed a couple of issues re. lack of KML support (#23 & #97). Does Fiona support KML? – user2856 Jan 26 '17 at 22:23
  • Thank you for spotting the from_features method. Saved my day! – jlandercy Apr 1 at 8:45
3

Since fiona.BytesCollection doesn't seem to work for TopoJSON here an solution that works for all without the need of gdal:

import fiona
import geopandas as gpd
import requests

# parse the topojson file into memory
request = requests.get('https://vega.github.io/vega-datasets/data/us-10m.json')
visz = fiona.ogrext.buffer_to_virtual_file(bytes(request.content))

# read the features from a fiona collection into a GeoDataFrame
with fiona.Collection(visz, driver='TopoJSON') as f:
    gdf = gpd.GeoDataFrame.from_features(f, crs=f.crs)
|improve this answer|||||
  • With geopandas==0.4.0, Fiona==1.8.4 and Python 3, I get DriverError: unsupported driver: 'TopoJSON'. – edesz Jan 20 '19 at 1:22
  • You are right. It was working till at least version 1.7.13 of Fiona – Mattijn Jan 20 '19 at 9:00
  • It is unfortunate that this doesn't work. I was trying to follow your example on GitHub for Altair choropleth plots but that too throws the exact same error at the line gdf = gpd.read_file(counties, driver='TopoJSON'). I thought that using with fiona.Collection... might work but sadly it doesn't. – edesz Jan 20 '19 at 17:20
  • @edesz this was a bug and will be fixed in Fiona 1.8.5, see: github.com/Toblerity/Fiona/issues/721 – Mattijn Feb 25 '19 at 21:25
2

Yes, it is possible now with Fiona (see https://github.com/Toblerity/Fiona/issues/409). I'm not sure if this feature is exposed yet in Geopandas.

|improve this answer|||||
2

When using Fiona 1.8, this can (must?) be done using that project's MemoryFile or ZipMemoryFile.

For example:

import fiona.io
import geopandas as gpd
import requests

response = requests.get('http://example.com/Some_shapefile.zip')
data_bytes = response.content

with fiona.io.ZipMemoryFile(data_bytes) as zip_memory_file:
    with zip_memory_file.open('Some_shapefile.shp') as collection:
      geodf = gpd.GeoDataFrame.from_features(collection, crs=collection.crs)
|improve this answer|||||
0

The easiest way is inputting the GeoJSON URL directly into gpd.read(). I'd tried extracting a shapefile from a zip before this using BytesIO & zipfile and had issues with gpd (specifically Fiona) accepting file-like objects.

import geopandas as gpd
import David.SQL_pull_by_placename as sql
import os

os.environ['PROJ_LIB'] = r'C:\Users\littlexsparkee\Anaconda3\Library\share\proj'

geojson_url = f'https://github.com/loganpowell/census-geojson/blob/master/GeoJSON/500k/2018/{sql.state}/block-group.json?raw=true'
census_tracts_gdf = gpd.read_file(geojson_url)
|improve this answer|||||
0

I prefer the result obtained by using the undocumented GeoDataFrame.from_features() rather than passing the GeoJSON to the GDF constructor directly:

import geopandas as gpd
import requests
data = requests.get("https://data.cityofnewyork.us/api/geospatial/arq3-7z49?method=export&format=GeoJSON")
gpd.GeoDataFrame().from_features(data.json())

Output

                       geometry                         name                                url           line objectid                                              notes
0    POINT (-73.99107 40.73005)                     Astor Pl  http://web.mta.info/nyct/service/  4-6-6 Express        1  4 nights, 6-all times, 6 Express-weekdays AM s...
1    POINT (-74.00019 40.71880)                     Canal St  http://web.mta.info/nyct/service/  4-6-6 Express        2  4 nights, 6-all times, 6 Express-weekdays AM s...
2    POINT (-73.98385 40.76173)                      50th St  http://web.mta.info/nyct/service/            1-2        3                              1-all times, 2-nights
3    POINT (-73.97500 40.68086)                    Bergen St  http://web.mta.info/nyct/service/          2-3-4        4           4-nights, 3-all other times, 2-all times
4    POINT (-73.89489 40.66471)             Pennsylvania Ave  http://web.mta.info/nyct/service/            3-4        5                        4-nights, 3-all other times

The resulting GeoDataFrame has the geometry column set correctly and all the columns as I would expect, without needing to unnest any FeatureCollections

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.