I've got a large time-series of rasters, derived from satellite images. Each raster is about 2000 x 2000 pixels, and I've got about 10,000 individual images. I'm trying to work out what the best way to store these is.

Of course, the best way to store them will depend on what I want to do with them - and in this case I want to be able to look at a time series for individual pixels, time-series averaged over specific areas (eg. vector administrative area data overlain on the raster), produce average images over various timescales, and view individual images.

The approach I've used before for shorter time series has just been to stack all of the images into a multi-band GeoTIFF - but I'm not even sure if a GeoTIFF will allow you to have 10,000 bands, or how well it will perform. An alternative would be to store them in thousands of separate files, but that would probably require a lot of custom programming work to be able to do the analysis.

I'm sure I can't be the first person to have this problem - what approaches are recommended for dealing with this sort of volume of time-series data?

Ideally I'd prefer solutions that work nicely with Python and don't require ArcGIS - but I'd be interested in any sensible ways forward.

| improve this question | | | | |
  • Have you looked at SciDB, rasdaman, PostGIS Rasters... – Spacedman Jun 6 '16 at 10:19
  • Ideally I was hoping for a 'simpler' solution than those (as this is on a project with a student), but that might be the best way forward – robintw Jun 6 '16 at 10:20
  • 1
    You might be able to create a virtual raster with 10,000 "bands" using the VRT driver in GDAL, rather than trying to smush 10,000 GeoTIFFs into one big file? Probably the simplest solution! – Spacedman Jun 6 '16 at 13:17
  • I'd work with them as NetCDF files in a THREDDS/TDS server, which would get you time series at pixels, and raster layers at particular timesteps. For averages over certain time scales, I'd use the NetCDF operators to do the averages to produce separate summary products. Geographical subsetting/aggregating might be harder, but the THREDDS server enables aggregation into one OpenDAP-enabled resource covering the three dimensional data-cube so you don't need to program against thousands of separate files. – Dave X Oct 11 '17 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.