4

I am trying to use AWS lambda to process some files stored in an S3 bucket using GDAL.

I have used information in the previous question enter link description here to test that I can read the file using /vsis3/

The difficulty has been to get GDAL working in AWS Lambda which is a Linux RHEL, which meant I have had to build from source.

I used the following config for that ./configure --disabled-shared --enable-static

Which gave me the executables that I can include in my code zip to upload to Lambda.

I have tested the GDAL executable using a local file within my code zip and that works, eg gdalinfo image.tif

However, trying to use

gdalinfo --config AWS_REGION "region" --config AWS_ACCESS_KEY_ID "KEY" --config AWS_SECRET_ACCESS_KEY "SECRET"/vsis3/bucket/key

causes an error message of

Unable to open datasource `/vsis3/bucket/key' with the following drivers.

Does anyone know a solution?

My fear is that /vsis3/ requires something that I am not using in the build process

EDITS

I managed to change my GDAL build to include curl (--with-curl) and have successfully used that from lambda to the S3 file using

gdalinfo "/vsicurl/image.tif"

But the vsis3 still fails

4
  • Is using rasterio an option? There are tutorials for rasterio lambda containers which seem pretty straightforward.
    – Kersten
    Feb 4 '17 at 16:28
  • @Kersten possibly although my NodeJS is better than my python, so will investigate rasterio further. Thanks for pointing me in that direction
    – tjmgis
    Feb 6 '17 at 20:45
  • Maybe you need to override CURL_CA_BUNDLE on the lambda? Grep cpl_vsi.h doc for more info.
    – Pete
    Jun 14 '17 at 21:16
  • @tjmgis, did you provide AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID configuration options? gdal.org/…
    – dr_times
    Jul 2 '18 at 11:59
1

AWS Lambda takes privileges to S3 service (including GDAL/OGR vsis3) from IAM role that runs the Lambda function. By default there is only privilege to run the Lambda function, but you can create your own IAM role having Lambda exec + S3 read privileges and assign it to execution role of your Lambda. Then you don't need to set AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID. The /vsicurl worked for you probably because you set public access to the source data on S3 but /vsis3 uses another mechanism...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.