6

I am building a system where we have a lot of large rasters (Sentinel-2 bands) stored in S3. A lot of this data is stored in our own buckets so we can store whatever format we find most usable.

I need to efficiently and often access small windows (often less than 500x500 pixels) from those rasters. Since this happens a lot I need to be able to download those small windows fast and without tranfering more data than i need.

For this i am using vsis3 which solves this problem in a nice way, just as explained in the answer here.

I am using GDAL 2.2.4 from conda-forge which seems to use OPENJPEG 2.3.0.

The problem

I see 2 choices of how i can store my data.

  1. As cloud optimized geotiffs
  2. As JPEG 2000

The nice properties of option 2 is that the filesizes are a lot smaller, and that all data which we have not processed on our own can be bulled directly from amazons own S3 bucket

However the problem with option 2 is that it seems to download much more data to access the window than if i am using option 1.

I can see that the jp2 files for sentinel-2 images by default has tilesize 1024. So i created a cloud optimized geotiff with a tilesize of 256. This performs much better (in terms of how much data i have to download), so i expected the tilesize to be the reason. However I then tried to make a cloud optimized geotiff with a tilesize of 1024 and again it performs much better than the .jp2 file.

Here is a very crude visualization of the data transfer for each file enter image description here

This show the data transfer required to fetch a 100x100 pixel window from a single band raster from each file type.

Now here is the question

Why do I have to download so much more data, when i try to access the same window from a jp2 file than from a geotiff file? It does not seem to be the tilesize, so what is the extra data i am downloading, and can i somehow avoid it?

I am just using the OpenJPEG driver, can this be the problem? and will a proprietary driver solve the problem i am describing?

Or do i simply have to bite the bullet and use cloud optimized geotiffs to access the windows faster, with the cost of some extra file size?

  • How do you create the JPEG2000 files and with what progression order? – user30184 May 3 '18 at 13:21
  • Currently I am actually not creating the JPEG2000 images, I am just consuming them. An example of the jp2 file i try to consume could be B04.jp2 available here: sentinel-s2-l1c.s3-website.eu-central-1.amazonaws.com/#tiles/32/… – the_cheff May 3 '18 at 13:33
  • 1
    It might be worth writing a mail to gdal-dev mailing list and ask how a Cloud optimised JPEG2000 file should be organised. For example the LRCP progression order in that Sentinel image may not be optimal for such use. – user30184 May 3 '18 at 14:16
  • 1
    It seems also that you should use a newest possible GDAL version which is compiled against OpenJPEG v. 2.3.0 or later erouault.blogspot.fi/2017/10/optimizing-jpeg2000-decoding.html. – user30184 May 3 '18 at 14:39
  • Cool. I am using GDAL 2.2.4 from conda-forge which seems to use OPENJPEG 2.3.0 so all should be good there. Thank you for the response. I do not have much knowledge on jp2, do you have any intuition of why the progression affects the amount of data i need to download? However due to the complexity of JP2000 I might be better off, sticking with geotiffs. I had just hoped that perhabs a quick setting change could help me. – the_cheff May 3 '18 at 17:22
3

Why do I have to download so much more data, when i try to access the same window from a jp2 file than from a geotiff file? It does not seem to be the tilesize, so what is the extra data i am downloading, and can i somehow avoid it?

As I understand it the JPEG2000 file layout is progressive - from lowest resolution to highest, and it's compression means that to read a single tile at full resolution you need to read through the progression for that tile. This means doing several file reads which on S3 means several HTTP requests — transferring more data and/or taking longer.

Cloud Optimised GeoTIFFs encode all metadata information in the file header for a single read, and getting a tile involves only one additional read/request. The CloudOptimizedGeoTIFF wiki page has more details. There's no relation between different overview levels so only the closest matching one needs to be read.

I am just using the OpenJPEG driver, can this be the problem? and will a proprietary driver solve the problem i am describing?

Kakadu is at least as bad in my experience, if not worse. One of the core GDAL developers who has developed the Cloud Optimised GeoTIFF approach has been working on OpenJPEG, so I suspect OpenJPEG will be the best optimised.

Or do i simply have to bite the bullet and use cloud optimized geotiffs to access the windows faster, with the cost of some extra file size?

Several compression methods are supported in TIFF files. If you need lossless compression, try the ZSTD support in newer GDAL versions. Otherwise, JPEG compression with YCbCr performs very well and file sizes will likely be similar to JP2. See the GeoTIFF format page for more details on available options. Tiles are all compressed independently. Remember to compress your overviews too!

  • Thanks, given this i believe Cloud optimized geotiffs are the way to go. I am using lossless compression, I think I tried LZW and Deflate with Predictor set to 2. It however does not create as small files as JP2. I recall something like 50% extra file size – the_cheff Oct 10 '18 at 12:02
  • 1
    ZSTD will do better than LZW/Deflate. Assuming you're the only one reading the files, it's not widely supported yet. Optimising the predictors & compression level can achieve quite a lot and is worth doing if your data is all quite similar. Note that you don't need to add overviews to your Cloud GeoTIFFs unless you're planning on reading areas at a lower resolution. They'll work just as efficiently for full-resolution sampling. FWIW, 4x levels of overviews will add ~35% to your original file size. – rcoup Oct 10 '18 at 16:05
  • Cool i will give ZSTD a shot then. I do need the overviews, however only in quite a lot lower resolution, so the extra filesize here is not that massive – the_cheff Oct 10 '18 at 17:38
  • You can have whatever levels of overview you need, so skipping straight to 1/16 or 1/64 or whatever will add negligible additional size (most of it is in 1/2 and 1/4) – rcoup Oct 18 '18 at 9:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.