5

I've used question on the topic as initial help and structured my code after it. I'm warping a raster file using these kwargs

rast = gdal.Open(/vsicurl/raster_url)
wkt = rast.GetProjection()
outbounds = [-180.00, -90.00, 180.00, 90.00]
warp_kwargs = {
                'format': 'vrt',
                'srcSRS': wkt,
                'dstSRS': 'EPSG:4326',
                'outputBounds': outbounds,
                'options':['SOURCE_EXTRA=1000','CENTER_LONG=0']
            }

gdal.Warp('/vsimem/rast_1.tif', rast, **warp_kwargs)

I then used this previous question to try to upload the file from /vsimem/ directly into an s3 bucket. I changed a couple things, so my code is as follows. (I use fp to mean filepath)

def upload_to_s3(mem_fp, s3_fp, bucket_name):
    f = gdal.VSIFOpenL(mem_fp, 'rb')
    gdal.VSIFSeekL(f, 0, 2)  # seek to end
    size = gdal.VSIFTellL(f)
    gdal.VSIFSeekL(f, 0, 0)  # seek to beginning
    data = gdal.VSIFReadL(1, size, f)
    gdal.VSIFCloseL(f)
    # Upload the raw data to s3
    s3 = boto3.client('s3', aws_access_key_id=ACCESS_KEY,
                      aws_secret_access_key=SECRET_KEY)
    s3.put_object(Key=s3_fp, Bucket=bucket_name, Body=data, ContentLength=size)

In this case size is the size of the raster and data is b'<VRTDataset rasterXSize="1440" rasterYSize="720" subClass="VRTWarpedDataset">. So this process does upload a file to s3, but it's only 3 kb when it should be around 8 mb. And when I try to download the file from s3 and plot it in QGIS, it errors out saying

ERROR 4: : No such file or directory
ERROR 5: GDALWarpOptions.Validate(): hSrcDS is not set.
gdalinfo failed - unable to open 'test.tif'.

I've tried using both s3.put_object() and s3.upload_file() and neither of them work. I've also tried using an s3.Object.put() which didn't work. I've tried putting the raster in /vsimem/temp which also didn't work. Not sure what else to try here.

7
  • 1
    Have you looked at that small file? Sounds like you uploaded just the VRT which is a text file describing geodata (and possible transformations). Jan 20, 2021 at 20:11
  • 1
    I'm not exactly sure how to look at it? What tools would I use? Also if that is the case, do you know how to upload the actual raster? would I just change the format in the warp kwargs to GTiff?
    – JWB
    Jan 20, 2021 at 20:22
  • Yep, that seemed to be it. Thank you! I did not realize that you could store GTiff in /vsimem/ I thought that was for VRTs only. You can submit as an answer if you want!
    – JWB
    Jan 20, 2021 at 20:25
  • 3
    In warp kwargs change 'format': 'vrt' to 'format': 'gtiff'. I would consider keeping the warp to VRT, then gdal.Translate to a vsimem tif with compression to reduce file size before uploading the tif.
    – user2856
    Jan 20, 2021 at 21:47
  • @user2856 thank you! If you want put it as an answer, I'll accept it.
    – JWB
    Jan 20, 2021 at 21:52

1 Answer 1

2

the issue here is within warp_kwargs. User2856 and bugmenot123 both pointed this out. Replace 'format':'vrt' with 'format':'GTiff'. The file that was getting uploaded to s3 was a vrt and not a tif. I was under the impression that only vrt files could be handled within the /vsimem/ directory; however, it works with the other gdal supported file formats!

After making that change, the correct raster file is being uploaded to s3.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.