Questions tagged [amazon-s3]
Amazon S3 (Simple Storage Service) is an online file storage web service offered by Amazon Web Services.
49 questions
0
votes
0
answers
31
views
Error when migrating gdal function proximity to aws s3
I am migrating a geoprocessing workflow to work on aws s3. I am having issues whith the function below. it first reads a geojson file, rasterize it, and then computes an euclidean distance raster. The ...
- 720
0
votes
0
answers
47
views
Unable to correctly visualize a PostGIS Out-DB COG Raster from private AWS S3 bucket on QGIS canvas
I cloud-optimized two similar TIFF files (small SRTM tiles) with GDAL and checked their validity as per here.
I created an AWS IAM account giving it read-only access on S3 buckets and generated access ...
- 345
0
votes
0
answers
75
views
Creating a single, large mosaic file in S3 from tiles that cannot fit into memory?
I'm trying to form a (very) large mosaic raster from constituent files in S3, and the amount of data is extremely large; I'm not sure on the actual size, but let's say it's really huge, say over a TB. ...
1
vote
0
answers
183
views
How to not check for pre-existing caches with mapproxy-seed?
The Swedish Geological Survey provides this WMS endpoint: https://resource.sgu.se/service/wms/130/jorddjupsmodell. I would like to create WMTS tiles from the SE.GOV.SGU.MISC.JORDDJUPSMODELL....
- 2,692
0
votes
1
answer
172
views
GDAL /vsicurl/ does not use range requests
I'm trying to extract a small part of a largish COG GeoTIFF using gdal_translate. The source file is stored on S3 and I have a presigned URL to it, and use /vsicurl/ to access it. For whatever reason, ...
- 1,093
3
votes
1
answer
227
views
Using COG images uploaded to Amazon S3 ( private bucket) with credentials in OpenLayers?
I am creating a GIS webApp and trying to show GeoTIFF images as basemap with OpenLayers. This approach works perfectly as long as the corresponding URL is publicly accessible, like this example: https:...
- 31
0
votes
1
answer
298
views
Slow initial loading of layers in Geoserver from huge COGs stored in a AWS S3
I am working with huge COGs (around 500GB) stored on my private S3 bucket. I am trying to publish these COGs as GeoTiff layers using COG extension in Geoserver. The Geoserver is deployed on an EC2 ...
- 83
1
vote
0
answers
79
views
Why are overviews of an out-db postgis raster not out-db?
I'm working on out-db postgis raster and have generated overviews, but why are the overviews generated of an out-db rasters stored in-db?
I have tried creating overviews both manually using ...
0
votes
1
answer
112
views
GeoServer s3 uri with ImagePyramid plugin
I would like to create pyramid and use the ImagePyramid plugin to create a store in GeoServer. I know GeoServer has a community module to support S3 uri in GeoTiff file, is it possible to use the S3 ...
1
vote
0
answers
321
views
COG file on S3 storage : Error when accessing COG with GeoServer cog-plugin
I configure a GeoServer's Image mosaics store to get GeoTIFF files from a private S3 storage,
compliant with AWS S3. The mosaic's already exists and I followed documentation for migration to S3 (https:...
- 11
0
votes
1
answer
1k
views
Using COG from private S3 bucket in geoserver
I am hosting some cloud optimised geotiffs in an S3 bucket to be used as a data store in geoserver. I am having trouble getting this working with a private bucket though, even after providing my AWS ...
- 77
2
votes
1
answer
703
views
Replacing GeoServer with COG and MVT from a bucket
Is it possible to completely replace GeoServer with a combination of Cloud Optimised GeoTIFFs and precreated Mapbox Vector Tiles hosted in cloud storage?
We are thinking to replace our datasets that ...
- 326
0
votes
1
answer
152
views
Authenticating against Earth Search API or STAC API to get raster images
I'm fairly new to the GIS domain and must query the Earth Search API to get some imagery. these endpoints do not require authentication:
https://earth-search.aws.element84.com/v0/search
https://sat-...
- 9
3
votes
2
answers
1k
views
"S3 Support for GeoTiff" vs "COG (Cloud Optimized GeoTIFF) Support" Geoserver Community Extensions
I have large raster datasets on AWS S3 (~100gb each) that I would like to serve as WMS through Geoserver.
Has the "S3 Support for GeoTiff" extension been replaced by the "COG" one?
...
- 326
0
votes
0
answers
221
views
Geoserver S3 Imagemosaic sidecar files and overviews
The geoserver documentation is a little bit off on that topic, but with this example was able to make the imageMosaic from S3 working with this example:
https://github.com/geotools/geotools/blob/...
1
vote
0
answers
343
views
Is there a way to improve tile performance for cloud-based storage?
I am trying to use cloud-storage for GWC tiles. However performance is abysmal, no matter the tweaks or provider I'm using.
What I've tried:
Providers: AWS, Wasabi
Access: S3 Plugin, File-system ...
- 324
0
votes
1
answer
667
views
Load raster from Requester-Pays S3 bucket into QGIS
I have successfully requested Landsat data from the USGS S3 bucket by following the provided tutorial. Now I want to pull a raster directly into QGIS via the Data Source Manager.
I have set my ...
- 111
1
vote
1
answer
943
views
Create STAC catalog file for my s3 bucket
I have s3 private bucket which contains .tif files.
Now, I need to create STAC file like the following:
https://earth-search.aws.element84.com/v0
https://earthengine-stac.storage.googleapis.com/...
- 77
1
vote
1
answer
1k
views
How do I add COGs from AWS S3 protocol to QGIS map canvas programmatically
I have a private bucket in S3 with COGs in it.
Following the instructions here, it's actually very simple to do it manually.
However, this does indeed only work manually.
Trying to add a COG with the ...
- 8,718
0
votes
0
answers
407
views
Download Sentinel-2 bands 04 and 08 from AWS S3 of a specific area
I am completely lost. I am trying to figure out where to start.
I need to develop a script that generates NDVI values from Sentinel images provided by AWS S3.
I've already found solutions for ...
- 111
4
votes
2
answers
1k
views
How to get Copernicus DEM GeoTIFFs for a bounding box using Python
Background
According to AWS's Copernicus Digital Elevation Model page,
The Copernicus DEM is a Digital Surface Model (DSM) which represents the surface of the Earth including buildings, ...
- 2,692
2
votes
0
answers
721
views
Configuring GeoServer on AWS to serve Cloud Optimized GeoTIFFs (COGs) from S3 [closed]
I want to use S3 buckets as data stores for ImageMosaics on GeoServer in AWS. I'm using the Support for S3 GeoTIFF and the ImageMosaic module. I'm using geoserver 2.16.
First of all, is this possible?...
- 49
1
vote
0
answers
684
views
Can Leaflet ZXY tiles be used in QGIS 3?
I have created tiles from a drone flight, loaded them onto Amazon S3 storage and used them in a leaflet web page served from an AWS EC2 instance. That is all working fine. I would my desktop QGIS ...
- 1,016
1
vote
0
answers
147
views
Unable to execute QGIS algorithm while taking Input raster directly from AWS S3 account
I have a raster file (.TIF) in my AWS S3 account. I am using GDAL to open that .TIF file and to save the reprojected .TIF file to my local computer using QGIS GDAL 'Warp (reproject)' algorithm as ...
- 729
6
votes
2
answers
3k
views
QGIS Read COG from S3
I am trying to read a Cloud Optimised GeoTiff (COG) from AWS S3. I can successfully run
gdalinfo /vsis3/cog-testing/HP_COG.tif
And see the details of the file and it says Layout: COG
But if I ...
- 3,330
0
votes
0
answers
109
views
Difference between S3GeoTiffReader and GeoTiffReader?
What's the difference between S3GeoTiffReader and GeoTiffReader provided by Geotools library?
I need to process a GeoTiff image retrieved from AWS S3 bucket.
- 1
2
votes
1
answer
1k
views
How to download tile from Amazon’s Sentinel-2 service
How do I access Sentinel-2 imagery via Amazon?
I'm new to GIS and have been relying on USGS earth explorer and EO Browser but would now like to access satellite imagery with higher resolution. I ...
- 449
4
votes
1
answer
1k
views
Write feature collection to S3
Is there a way to write a feature collection from Python memory to an S3 bucket (as Esri Shapefile, zipped Shapefile or something else) without writing it first to disk?
I tried writing to a ...
- 2,302
3
votes
1
answer
7k
views
Is it possible to serve vector tiles directly from S3?
I have seen that it is possible to use S3 as a storage backend for vector tile servers such as Tegola. These work by fetching the tile from S3, then serving it through an application server.
Is there ...
- 5,710
3
votes
1
answer
3k
views
Clipping/cropping/masking a raster tile layer in Mapbox GL JS
I'm loading Landsat tiles into a Mapbox GL JS map via an AWS S3 tile server, and this is working OK, but what I'd really like to do is clip the tile layer to a GeoJSON polygon which is loaded in ...
- 574
1
vote
3
answers
2k
views
How to download Sentinel2 L1C using AWS Python SDK (boto3)
The permissions for Sentinel2 L1C changed today
https://www.sentinel-hub.com/faq/how-will-access-permission-changes-aws-bucket-affect-users-accessing-data-directly-non-sentinel#
How can I send a '...
- 310
0
votes
0
answers
453
views
Sentinel-1 GRD and Sentinel-2 L1C on AWS bucket policy
Recently I started to work with Sentinel-1 data hosted on AWS S3 bucket 'sentinel-s1-l1c'. The bucket is requester pays and requires AWS signature version 4. I need to access the files via HTTP GET ...
- 938
2
votes
0
answers
435
views
How can I create a RasterLayer object layers from a list of files stored as data frame?
I am trying to load raster files stored in AWS s3 bucket and I've managed to read them as data frame. How can I change them to rasterlayer objects?
library(aws.3)
library(raster)
library(rgdal)
#...
- 53
9
votes
1
answer
3k
views
Should i use jp2 or cloud optimized geotiffs to efficiently access small windows from Rasters on S3
I am building a system where we have a lot of large rasters (Sentinel-2 bands) stored in S3. A lot of this data is stored in our own buckets so we can store whatever format we find most usable.
I ...
- 241
0
votes
1
answer
3k
views
How to import images bucket from AWS S3 using python
How to import images bucket from AWS s3 using python without declaring LOCAL_PATH, i want total bucket images in python environment in array format, from one week on words i am facing this issue, ...
- 1
3
votes
1
answer
2k
views
Import point data from amazon s3 into QGIS
I'm working with spatial/temporal data, stored in s3 (AWS). The data is in the format:
ID, Date, Lat, Lon
23424 2017-02-03T23:15:15Z 23.434 -5.2343
23424 2017-02-03T23:20:...
- 565
6
votes
0
answers
876
views
Disable readdir on open in gdal_translate when using /vsicurl/
I'm using /vsicurl/ file handler to read Landsat 8 data out of S3. Following GDAL documentation "to minimize the total number of HTTP requests outside of the target GeoTIFF file" I set ...
- 938
12
votes
4
answers
1k
views
Carrying out atmospheric correction on Sentinel-2 data from Amazon Web Services?
I'm wondering what the best method is for correcting Sentinel-2 images downloaded in JP2 format (from Amazon Web Services - AWS) to BOA reflectance? As the files downloaded from the sentinel scihub in ...
- 131
28
votes
6
answers
31k
views
How to efficiently access files with GDAL from an S3 bucket using VSIS3?
So, GDAL has recently added a new feature that allows random reading of S3 bucket files. I am looking to crop GDAL images from multiple tiles of an image without having to download the whole file. I'...
- 529
1
vote
0
answers
293
views
Process Landsat8 data in AWS S3 from EC2 instance
I'm wondering what is the best way of processing huge amounts of Landsat8 images stored in the AWS S3 buckets from an Ec2 instance located in the same availability zone.
Should I download the scenes ...
- 11
1
vote
0
answers
175
views
Import postgis layer from S3 link
I have a postgis layer (layer.sql), which I am trying to add to my postgres database (mydatabase). The postgis layer is stored on S3 (https://s3.us.../layer.sql). I can add the layer to my database ...
- 11
2
votes
0
answers
477
views
How to access Landsat bucket from Amazon?
I tried to access amazon Landsat S3 bucket with the following python code
import boto
import boto.s3.connection
from boto.s3.connection import S3Connection
access_key = <userid>
secret_key = &...
- 607
-1
votes
2
answers
2k
views
How to read Landsat images from Amazon S3 using Python? [closed]
I would like to read and process Landsat images from Amazon S3 using Python
- 607
15
votes
2
answers
7k
views
Geoserver with Amazon s3 bucket
I have Apache running Geoserver on my virtual instance, and I am wondering whether the Geoserver data directory can point to an Amazon S3 bucket. Is this possible? How does this affect performance?
...
- 356
1
vote
1
answer
169
views
Free software to publish tiles on Amazon S3?
Does anyone know of free software that would allow me to publish tiles on Amazon S3? I know of Arc2Earth, but it is not free.
- 191
9
votes
3
answers
13k
views
What is the best way to host mbtiles?
What is/are the best way(s) to host mbtiles?
I have tried:
hosting on my own server with tileserver.php: this works and is very straighforward and simple to set up, but doesn't seem to be super fast ...
- 347
1
vote
0
answers
2k
views
Access-Control-Allow-Origin problem when loading json file from Amazon S3 to display in OpenLayers
This is my code for displaying geojson in OpenLayers, where "geourl" is directed to a json file in Amazon S3.
geojson_layer = new OpenLayers.Layer.Vector("GeoJSON", {
strategies: [new OpenLayers....
- 511
1
vote
1
answer
316
views
Syncing private S3 data with CartoDB?
I've been using the new Synced Table feature of CartoDB, and it's working great.
But some of the data I want to sync with is not public data. The data provider keeps it on S3, but requires ...
- 730
2
votes
1
answer
211
views
Is it possible to store data on Amazon S3 for use in OpenGeo Suite?
We've got about 500GB of geospatial data we'd like to "web-enable" to various degrees and OpenGeo Suite seems like a good route to go down given our circumstances. We can sort out a server (probably ...
- 1,002