I'm having trouble locating any discussion or comparative benchmarking of different raster file formats (e.g., for use in data analysis in R). Does anybody have any insight into why particular formats may be faster or slower? Or should the differences be minimal?
Specifically, I'm interested in if converting a raster (e.g., a GEOTIFF file) to a different format (e.g, a netCDF) is ever worthwhile for the purpose of speeding read/write and other operations.
Here's an old blog article of mine looking at the file size and access time of the formats. I didn't investigate the write speed, only the access time. I'd say they would probably be directly related, but wouldn't be able to vouch for it.
Article Summary: It seems that Packbits gives you the best access times (at expense of disk space), whereas Deflate gives you intermediate/slow access times for intermediate/small files. Also, you can test access times more empirically by creating thumbnails of various sizes and timing how long it takes. Example command: time gdal_translate -outsize <thumbnail dimensions> -of GTiff <compressed image file> <thumbnail file>
Assuming that the only thing relevant to R in this case is how quickly it can read data from the file, just as any other process would, then that should give you a good indication.
For read/write operations, you can test the speed of those operations using system.time(). Here are some results from loading a DEM file in R (Raster package) translated into four formats (ASCII, IMG and TIF with no compression and Deflate). For example, on a ~26MB raster:
> system.time(dem <- raster("~/workspace/TEMP/kideporegion.img"))
user system elapsed
0.154 0.002 0.157
'Elapsed' gives the total time (seconds) taken for the operation. Running the operations 10 times each and looking at mean elapsed time:
mean diff
ASC 0.2237 0.3317
IMG 0.1544 0.0318
tif-Deflate 0.1510 0.0099
tif-none 0.1495 0.0000
tif-pack 0.1513 0.0118
TIFF with no compression is the fastest ... followed very closely by Deflate (0.1% slower) and TIFF-Packbits (1.8% slower), then IMG (3.2% slower) and ASC (33% slower). (This is on a Macbook Pro 2.4 GHz with an SSD, so fast disk operations)
This is simply to load the files, not manipulate them.
Maybe it is really not a question of which raster image format has better opening benchmarks--rather which raster image formats are most efficient raster source formats for opening and reading as input into an R numerical array. And subsequently--what is the most efficient output format from R assuming you'd be outputting results back to raster.
Either way, if you're going to work with raster in R you will likely be using the rgdal and R ncdf packages to supplement what is contained in the R raster package. With principal reliance on the gdalwarp command. Need to work out format dependencies there to make your raster choice. You'll find a fair bit of coverage on SO and various OSGEO and R forums/blogs/wiki.
But as this is a GIS forum where Python use is in relative ascendency, I'll note that there are advantages to working with raster data in a Python numpy array, with similar dependence on the gdal libraries for raster loading, conversion and export. Some folks find memory management and code structure in Python preferable over native R--perhaps take a look at RPy2 or PypeR as either may be appropriate for your analysis use.
A big question is whether you are going to read the entire raster from the file into memory before processing it, or whether the file is so large that you will process it incrementally, or process some subset of the overall file.
If you will load it all into memory, then you will be doing mostly sequential access, and the fastest format will be a tossup between plain and compressed storage (depending on things like how fast your CPU is versus disk). Any of the binary file formats will probably be pretty close (ASCII will be slower).
If you need to process a subset of a very large file, then a format which groups the subset you want closer together may be faster - eg: tiles or a format which can compute offsets. Sometimes uncompressed approaches gain here because it may be trivial to compute where any given part of the image resides within the file, especially if you need only part of a very large row, but compression can be done in a granular fashion that works well for some access patterns.
Sorry, but you'll probably have to benchmark depending on your access pattern, rather than getting a one-size-fits-all. It may of course depend not just on the file format and the above factors, but on the drivers for that format and your software.
The way you think about these kinds of problems is in terms of how your application accesses your file, vs. how the data is laid out in your file. The idea is that if you can access your data sequentially, it will be much more efficient than if you access it randomly.
GeoTIFF is a collection of 2D "images" or arrays. NetCDF is a general purpose storage for multidimensional arrays. But if you store the arrays in the same way in netCDF as they are in GeoTIFF, you will get the same performance, more or less.
One can also rearrange the data in netCDF, so in principle could optimize for your reading patterns. My guess is that most GIS application are optimized for the GeoTIFF 2D layout, so theres not much to gain by rearranging.
Finally, Id say it really only matters when you have very large files, at least tens of megabytes.
I read a couple of pages about this several years ago and have since used tiff with packbits compression, tiled with a geotiff header, and have been happy.
But after reading the following, I will reconsider and maybe use the deflate variety.
So many packages use GDAL under the hood, e.g., rgdal, QGIS, GRASS, etc. If I was using one of those packages, then I would think of converting my images to vrt. I've often seen it recommended that when you need to use two GDAL commands, then the intermediate file should be a vrt file because the read overhead is minimal (e.g., http://www.perrygeo.com/lazy-raster-processing-with-gdal-vrts.html). It sounds like your workflow is: convert once and read many times. Maybe vrt would be appropriate.
[Edit: link adjusted]
