1

This question is somewhat similar to this one: How to efficiently access files with GDAL from an S3 bucket using VSIS3?, except I am trying to access bucket files from Google Cloud Storage, with Python. I am using GDAL 2.3.1, so I should be able to use the VSIGS virtual driver.

According the the example I came across, it looks like this simple piece of code should work:

from osgeo import gdal

ds = gdal.Open('/vsigs/my_bucket/image.tif') # doesn't work
ds = gdal.Open('gs://my_bucket/image.tif') # doesn't work either

But I keep getting a "file not found" error, so it looks like GDAL does not understand that I'm trying to open a GCS file. What am I missing?

Notes:

  • gcloud is properly installed and configured on my computer (the command gsutil ls gs://earthengine-public/ works properly).
  • I'm working inside a Python virtual environment, so that might be the issue there.
  • You can find Python examples from GDAL autotests github.com/OSGeo/gdal/blob/master/autotest/gcore/vsigs.py. – user30184 Aug 5 '18 at 19:37
  • @user30184 I tried these examples but it didn't work for me. I really think it has something to do with the virtual environment. Anyway, I found a workaround using rasterio instead of GDAL, so my question is obsolete now. – ArMoraer Aug 7 '18 at 12:00
  • Can you share a lead on or example of what you found on vsigs with rasterio? Looks like it is an active issue and area of discussion. – metasequoia Aug 10 '18 at 1:20
1

Following @metasequoia's suggestion, I'm posting the workaround I found for this issue. This doesn't actually answer the question since it relies on rasterio instead of GDAL, but it might help other people.

This minimal example worked for me (with rasterio 1.0.1):

import rasterio

gs_access_key= "..."
gs_secret_access_key = "..."
url = 'bucket/image.tif'

with rasterio.Env(GS_SECRET_ACCESS_KEY=gs_secret_access_key, GS_ACCESS_KEY_ID=gs_access_key):
    with rasterio.open('/vsigs/{}'.format(url)) as src:
        print(src.width, src.height)
  • Thanks, that helped me out when looking for a solution to read tifs in google cloud storage from compute nodes (without data duplication). – metasequoia Dec 4 '18 at 0:17
0

Rasterio added support for gcs urls in version 1.0.15. If you're working with a recent version you can now use the following:

import rasterio
import os

#can also set these as a normal env var outside of python
os.environ['GS_SECRET_ACCESS_KEY'] = ''
os.environ['GS_ACCESS_KEY_ID'] = ''

#url should look like gs://...
with rasterio.open(url) as src:
    print(src.width, src.height)

This is convenient, as you no longer need to wrap rasterio requests in the env.

You can also authenticate through other gcs methods (e.g. set GOOGLE_APPLICATION_CREDENTIALS variable to service account credentials json), however, these methods currently don't work unless you've installed rasterio from a source distribution (pip install rasterio --no-binary rasterio) due to gdal version incompatibilities.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.