Read files with Python GDAL using VSIGS

Question

This question is somewhat similar to this one: How to efficiently access files with GDAL from an S3 bucket using VSIS3?, except I am trying to access bucket files from Google Cloud Storage, with Python. I am using GDAL 2.3.1, so I should be able to use the VSIGS virtual driver.

According the the example I came across, it looks like this simple piece of code should work:

from osgeo import gdal

ds = gdal.Open('/vsigs/my_bucket/image.tif') # doesn't work
ds = gdal.Open('gs://my_bucket/image.tif') # doesn't work either

But I keep getting a "file not found" error, so it looks like GDAL does not understand that I'm trying to open a GCS file. What am I missing?

Notes:

• gcloud is properly installed and configured on my computer (the command gsutil ls gs://earthengine-public/ works properly).
• I'm working inside a Python virtual environment, so that might be the issue there.

Answer 1

Rasterio added support for gcs urls in version 1.0.15. If you're working with a recent version you can now use the following:

import rasterio
import os

#can also set these as a normal env var outside of python
os.environ['GS_SECRET_ACCESS_KEY'] = ''
os.environ['GS_ACCESS_KEY_ID'] = ''

#url should look like gs://...
with rasterio.open(url) as src:
    print(src.width, src.height)

This is convenient, as you no longer need to wrap rasterio requests in the env.

You can also authenticate through other gcs methods (e.g. set GOOGLE_APPLICATION_CREDENTIALS variable to service account credentials json), however, these methods currently don't work unless you've installed rasterio from a source distribution (pip install rasterio --no-binary rasterio) due to gdal version incompatibilities.

Answer 2

Following @metasequoia's suggestion, I'm posting the workaround I found for this issue. This doesn't actually answer the question since it relies on rasterio instead of GDAL, but it might help other people.

This minimal example worked for me (with rasterio 1.0.1):

import rasterio

gs_access_key= "..."
gs_secret_access_key = "..."
url = 'bucket/image.tif'

with rasterio.Env(GS_SECRET_ACCESS_KEY=gs_secret_access_key, GS_ACCESS_KEY_ID=gs_access_key):
    with rasterio.open('/vsigs/{}'.format(url)) as src:
        print(src.width, src.height)