4

I am trying to use AWS lambda to process some files stored in an S3 bucket using GDAL.

I have used information in the previous question enter link description here to test that I can read the file using /vsis3/

The difficulty has been to get GDAL working in AWS Lambda which is a Linux RHEL, which meant I have had to build from source.

I used the following config for that ./configure --disabled-shared --enable-static

Which gave me the executables that I can include in my code zip to upload to Lambda.

I have tested the GDAL executable using a local file within my code zip and that works, eg gdalinfo image.tif

However, trying to use 

gdalinfo --config AWS_REGION "region" --config AWS_ACCESS_KEY_ID "KEY" --config AWS_SECRET_ACCESS_KEY "SECRET"/vsis3/bucket/key

causes an error message of

Unable to open datasource `/vsis3/bucket/key' with the following drivers.

Does anyone know a solution?

My fear is that /vsis3/ requires something that I am not using in the build process

EDITS

I managed to change my GDAL build to include curl (--with-curl) and have successfully used that from lambda to the S3 file using

gdalinfo "/vsicurl/image.tif"

But the vsis3 still fails

Improve this question
4
  • Is using rasterio an option? There are tutorials for rasterio lambda containers which seem pretty straightforward.
    – Kersten
    Commented Feb 4, 2017 at 16:28
  • @Kersten possibly although my NodeJS is better than my python, so will investigate rasterio further. Thanks for pointing me in that direction
    – tjmgis
    Commented Feb 6, 2017 at 20:45
  • Maybe you need to override CURL_CA_BUNDLE on the lambda? Grep cpl_vsi.h doc for more info.
    – Pete
    Commented Jun 14, 2017 at 21:16
  • @tjmgis, did you provide AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID configuration options? gdal.org/…
    – dr_times
    Commented Jul 2, 2018 at 11:59

1 Answer 1

Reset to default
1

AWS Lambda takes privileges to S3 service (including GDAL/OGR vsis3) from IAM role that runs the Lambda function. By default there is only privilege to run the Lambda function, but you can create your own IAM role having Lambda exec + S3 read privileges and assign it to execution role of your Lambda. Then you don't need to set AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID. The /vsicurl worked for you probably because you set public access to the source data on S3 but /vsis3 uses another mechanism...

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.