Skip to main content
27 votes
Accepted

How to efficiently access files with GDAL from an S3 bucket using VSIS3?

I've found when something isn't particularly well documented in GDAL, that looking through their tests can be useful. The /vsis3 test module has some simple examples, though it doesn't have any ...
user2856's user avatar
  • 66.6k
11 votes

How to efficiently access files with GDAL from an S3 bucket using VSIS3?

Since /vsis3/ is implemented in GDAL you can also use rasterio to read Windows of S3 datasets. This requires either your credentials to be set up for boto or using rasterios AWS session handler. ...
Kersten's user avatar
  • 9,919
8 votes
Accepted

Should i use jp2 or cloud optimized geotiffs to efficiently access small windows from Rasters on S3

Why do I have to download so much more data, when i try to access the same window from a jp2 file than from a geotiff file? It does not seem to be the tilesize, so what is the extra data i am ...
rcoup's user avatar
  • 751
7 votes
Accepted

Is it possible to serve vector tiles directly from S3?

It is possible. I recommend you to use Route 53, S3, CloudFront. Host your domain name on AWS via AWS Route 53 DNS (yourdomain.com) to create a public zone. Create a SSL Cert for your domain name via ...
Kyros Koh's user avatar
  • 407
7 votes
Accepted

How to download Sentinel2 L1C using AWS Python SDK (boto3)

Here is a sample code that should work (given that your config and credentials are saved in ~/.aws) import boto3 s3_client = boto3.Session().client('s3') response = s3_client.get_object(Bucket='...
AleksMat's user avatar
  • 416
5 votes

Replacing GeoServer with COG and MVT from a bucket

Yes, but it'll be some work, and you need the level of control having your own application brings. If you wanted to make it work with a webpage, you'd need a stateless serving layer between S3 and the ...
Paul Norman's user avatar
  • 1,596
4 votes

How to efficiently access files with GDAL from an S3 bucket using VSIS3?

Try using an XML file to store the WMS info in, more details are at the GDAL WMS documentation. Here's an example WMS XML file to retrieve data from Mapzen's Elevation API: <GDAL_WMS> <...
clhenrick's user avatar
  • 1,845
4 votes

What is the best way to host mbtiles?

7 years later, there are now other formats similar to MBtiles (i.e. single-file, supporting raster & tile, GIS, leaflet...) that are "cloud-optimized": Cloud-optimised Geo-Tiff (2018) (...
eddygeek's user avatar
  • 165
4 votes
Accepted

How do I add COGs from AWS S3 protocol to QGIS map canvas programmatically

You can add S3 using the following bucket_name = 'your_bucket_name' bucket_path_to_tif = 'subdir/myfile.tif' rlayer = QgsRasterLayer(f'/vsis3/{bucket_name}/{bucket_path_to_tif}', "My layer name&...
ThomasG77's user avatar
  • 30.9k
3 votes

How to get Copernicus DEM GeoTIFFs for a bounding box using Python

Try our open source python package available through PyPI or conda-forge that merges tiles from the open AWS registry of Copernicus 30 meter DEM. (Obviously: I am one of the library developers). We ...
Charlie's user avatar
  • 31
3 votes
Accepted

Create STAC catalog file for my s3 bucket

Great question. The easiest way to do this is using RIO STAC, which was developed by the magnificent Vincent Sarago. I use it programmatically (from Python). Make sure you enable the Projection ...
Alex Leith's user avatar
  • 13.5k
3 votes

QGIS Read COG from S3

From the same Data Source Manager, in Raster tab, I have selected in the combobox : HTTP/HTTPS/FTP and as URI : https://cog-testing.s3-eu-west-1.amazonaws.com/HP_COG.tif And it works on QGIS 3.8.3 ...
J. Monticolo's user avatar
  • 15.7k
2 votes
Accepted

Import point data from amazon s3 into QGIS

This question/answer explains how to use a .vrt file to create a link to the URL that corresponds to the S3 file/bucket. This assumes that you are trying to read the entire S3 file, not a subset of it ...
Llaves's user avatar
  • 5,973
2 votes

"S3 Support for GeoTiff" vs "COG (Cloud Optimized GeoTIFF) Support" Geoserver Community Extensions

As Ian said, both still exist. The general S3 one has seen no development in the last few years, while the COG one is being improved regularly. However, the old one still has a few advantages for ...
Andrea Aime's user avatar
  • 17.5k
2 votes
Accepted

"S3 Support for GeoTiff" vs "COG (Cloud Optimized GeoTIFF) Support" Geoserver Community Extensions

No, both extensions continue to exist. COG is probably faster (and cheaper) that the general S3 but may require you to reprocess your tiffs to make them COGs. Just to note that as they are community ...
Ian Turton's user avatar
  • 81.8k
1 vote

GeoServer s3 uri with ImagePyramid plugin

I tried to configure both ImagePyramid plugin and S3GeoTiff community module and I found it isn't supported the s3 uri provided by community module also in the pyramid plugin :(
Federico Paparoni's user avatar
1 vote

Using COG from private S3 bucket in geoserver

The answer was simple: set permissions on the bucket. { "Version": "2012-10-17", "Statement": [ { "Sid": "AddPerm", "Effect&...
sobmortin's user avatar
1 vote

Authenticating against Earth Search API or STAC API to get raster images

Sentinel 2 on AWS is requester pays. In general, bucket owners pay for all Amazon S3 storage and data transfer costs that are associated with their bucket. However, you can configure a bucket to be a ...
user2856's user avatar
  • 66.6k
1 vote
Accepted

Load raster from Requester-Pays S3 bucket into QGIS

I was able to achieve this by setting the Gdal environment variable AWS_REQUEST_PAYER=requester in the same place as the others above.
Alecg_O's user avatar
  • 111
1 vote
Accepted

How to get Copernicus DEM GeoTIFFs for a bounding box using Python

From the boto3 documentation, here's how you download a file: s3 = boto3.client('s3') s3.download_file('BUCKET_NAME', 'OBJECT_NAME', 'FILE_NAME') In your example it might look like this key = objects[...
bckygldstn's user avatar
1 vote

QGIS Read COG from S3

What normally works for me is: Verify you have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set in either Project menu > Properties, Variables tab OR QGIS Settings, Variables tab. These values you ...
Nitay's user avatar
  • 141
1 vote

How to download tile from Amazon’s Sentinel-2 service

The s3 Link for your sentinel data is the link to an Amazon S3 bucket. That is where the data is stored. Amazon's cloud services can be confusing for beginners and I am far from an expert. You can ...
Chris P's user avatar
  • 1,639
1 vote

Write feature collection to S3

ZipMemoryFile is read only so far. https://fiona.readthedocs.io/en/latest/fiona.html#fiona.io.ZipMemoryFile
Kai's user avatar
  • 11
1 vote

Clipping/cropping/masking a raster tile layer in Mapbox GL JS

Use the raster source's bounds: https://www.mapbox.com/mapbox-gl-js/style-spec#sources-raster-bounds "landsat": { "type": "raster", "url": "https://some-s3-bucket/{z}/{x}/{y}.png", "...
alphabetasoup's user avatar
1 vote

How to download Sentinel2 L1C using AWS Python SDK (boto3)

Another alternative is to set the ExtraArgs argument for the download_file function, as follows: bucket_name = 'sentinel-s2-l1c' key = 'tiles/7/W/FR/2018/3/31/0/B01.jp2' file_name = 'B01.jp2' s3 = ...
jnyborg's user avatar
  • 11
1 vote

How to import images bucket from AWS S3 using python

Well not sure what is the exact error you are facing. Please post the log. I am sure you would have solved this by now. just to help others. we should do something like this: bucket = conn.Bucket('...
Nagashayan's user avatar
1 vote

Carrying out atmospheric correction on Sentinel-2 data from Amazon Web Services?

Some L2A images (BOA reflectance, with cloud mask) can now be downloaded from THEIA. They are produced with MAJA (not SEN2COR) and cover western Europe and some spots elsewhere.
mdag02's user avatar
  • 306
1 vote

Carrying out atmospheric correction on Sentinel-2 data from Amazon Web Services?

Some BOA products now may be downloaded directly from a S3 bucket of L2A data: http://sentinel-pds.s3-website.eu-central-1.amazonaws.com/ s3://sentinel-s2-l2a in EU Central region for L2A data ...
Fred Liporace's user avatar
1 vote

How to efficiently access files with GDAL from an S3 bucket using VSIS3?

To read from S3, I used '/vsis3_streaming/ (AWS S3 files: streaming)'. (https://gdal.org/user/virtual_file_systems.html#vsis3-streaming-aws-s3-files-streaming). # first set s3 client session = boto3....
Rafa's user avatar
  • 111
1 vote

How to read Landsat images from Amazon S3 using Python?

You should be able to read Landsat directly out of S3 using /vsicurl/ file handler. Before coding it up in Python, check if you can access a file from the commandline. I think /vsicurl/ may work in ...
Pete's user avatar
  • 343

Only top scored, non community-wiki answers of a minimum length are eligible