Sign up ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I'm having trouble locating any discussion or comparative benchmarking of different raster file formats (e.g., for use in data analysis in R). Does anybody have any insight into why particular formats may be faster or slower? Or should the differences be minimal?

Specifically, I'm interested in if converting a raster (e.g., a GEOTIFF file) to a different format (e.g, a netCDF) is ever worthwhile for the purpose of speeding read/write and other operations.

share|improve this question
This question is relevant to GIS, but I suspect you're more likely to get answers on SO, which has a strong subcommunity of R experts. If you don't get an answer quickly, please just flag this question and a moderator will migrate it for you. – whuber Sep 2 '11 at 15:52

7 Answers 7

Here's an old blog article of mine looking at the file size and access time of the formats. I didn't investigate the write speed, only the access time. I'd say they would probably be directly related, but wouldn't be able to vouch for it.

Article Summary: It seems that Packbits gives you the best access times (at expense of disk space), whereas Deflate gives you intermediate/slow access times for intermediate/small files. Also, you can test access times more empirically by creating thumbnails of various sizes and timing how long it takes. Example command: time gdal_translate -outsize <thumbnail dimensions> -of GTiff <compressed image file> <thumbnail file>

Assuming that the only thing relevant to R in this case is how quickly it can read data from the file, just as any other process would, then that should give you a good indication.

share|improve this answer
+1 for the linked article, but the important info is offsite and will be lost to us if that page ever goes down or moves. I suggest giving a summary conclusion of the article so that in the event the page is not available, even momentarily, readers have something to work with for future research and thinking. Thanks! – matt wilkie Sep 6 '11 at 19:22
Fair enough! It seems that Packbits gives you the best access times (at expense of disk space), whereas Deflate gives you intermediate/slow access times for intermediate/small files. Also, you can test access times more empirically by creating thumbnails of various sizes and timing how long it takes. Example command: "time gdal_translate -outsize <thumbnail dimensions> -of GTiff <compressed image file> <thumbnail file>" – R Thiede Sep 7 '11 at 6:21
thanks! I folded the summary into the answer itself, so it's more self contained (see edit link at bottom left of each answer/question). – matt wilkie Sep 7 '11 at 18:57
Oh, right. Thanks ^.^; – R Thiede Sep 8 '11 at 8:49

For read/write operations, you can test the speed of those operations using system.time(). Here are some results from loading a DEM file in R (Raster package) translated into four formats (ASCII, IMG and TIF with no compression and Deflate). For example, on a ~26MB raster:

> system.time(dem <- raster("~/workspace/TEMP/kideporegion.img"))
 user  system elapsed 
 0.154   0.002   0.157 

'Elapsed' gives the total time (seconds) taken for the operation. Running the operations 10 times each and looking at mean elapsed time:

              mean   diff
ASC         0.2237 0.3317
IMG         0.1544 0.0318
tif-Deflate 0.1510 0.0099
tif-none    0.1495 0.0000
tif-pack    0.1513 0.0118

TIFF with no compression is the fastest ... followed very closely by Deflate (0.1% slower) and TIFF-Packbits (1.8% slower), then IMG (3.2% slower) and ASC (33% slower). (This is on a Macbook Pro 2.4 GHz with an SSD, so fast disk operations)

This is simply to load the files, not manipulate them.

share|improve this answer

Maybe it is really not a question of which raster image format has better opening benchmarks--rather which raster image formats are most efficient raster source formats for opening and reading as input into an R numerical array. And subsequently--what is the most efficient output format from R assuming you'd be outputting results back to raster.

Either way, if you're going to work with raster in R you will likely be using the rgdal and R ncdf packages to supplement what is contained in the R raster package. With principal reliance on the gdalwarp command. Need to work out format dependencies there to make your raster choice. You'll find a fair bit of coverage on SO and various OSGEO and R forums/blogs/wiki.

But as this is a GIS forum where Python use is in relative ascendency, I'll note that there are advantages to working with raster data in a Python numpy array, with similar dependence on the gdal libraries for raster loading, conversion and export. Some folks find memory management and code structure in Python preferable over native R--perhaps take a look at RPy2 or PypeR as either may be appropriate for your analysis use.

share|improve this answer
For manipulating netCDF data (raster source or otherwise) in R, here are links to two R CRAN hosted netCDF packages, ncdf4 -- and RNetCDF -- – V Stuart Foote Sep 4 '11 at 15:17

The way you think about these kinds of problems is in terms of how your application accesses your file, vs. how the data is laid out in your file. The idea is that if you can access your data sequentially, it will be much more efficient than if you access it randomly.

GeoTIFF is a collection of 2D "images" or arrays. NetCDF is a general purpose storage for multidimensional arrays. But if you store the arrays in the same way in netCDF as they are in GeoTIFF, you will get the same performance, more or less.

One can also rearrange the data in netCDF, so in principle could optimize for your reading patterns. My guess is that most GIS application are optimized for the GeoTIFF 2D layout, so theres not much to gain by rearranging.

Finally, Id say it really only matters when you have very large files, at least tens of megabytes.

share|improve this answer
+1 for the point that random access, or read of arbitrary location, is very different from a sequential one after the other until the whole file is done read. I may be off base, but I think Geotiff also supports tiled-storage and arbitrary access, it's just that by strip/row is the most common and widely supported. Also these days "very large files" in GIS tend to be in the multi GB range. ;-) – matt wilkie Dec 8 '11 at 22:33

A big question is whether you are going to read the entire raster from the file into memory before processing it, or whether the file is so large that you will process it incrementally, or process some subset of the overall file.

If you will load it all into memory, then you will be doing mostly sequential access, and the fastest format will be a tossup between plain and compressed storage (depending on things like how fast your CPU is versus disk). Any of the binary file formats will probably be pretty close (ASCII will be slower).

If you need to process a subset of a very large file, then a format which groups the subset you want closer together may be faster - eg: tiles or a format which can compute offsets. Sometimes uncompressed approaches gain here because it may be trivial to compute where any given part of the image resides within the file, especially if you need only part of a very large row, but compression can be done in a granular fashion that works well for some access patterns.

Sorry, but you'll probably have to benchmark depending on your access pattern, rather than getting a one-size-fits-all. It may of course depend not just on the file format and the above factors, but on the drivers for that format and your software.

share|improve this answer

I read a couple of pages about this several years ago and have since used tiff with packbits compression, tiled with a geotiff header, and have been happy.

arcpad team article


But after reading the following, I will reconsider and maybe use the deflate variety.

Arcpad site

share|improve this answer

So many packages use GDAL under the hood, e.g., rgdal, QGIS, GRASS, etc. If I was using one of those packages, then I would think of converting my images to vrt. I've often seen it recommended that when you need to use two GDAL commands, then the intermediate file should be a vrt file because the read overhead is minimal (e.g., It sounds like your workflow is: convert once and read many times. Maybe vrt would be appropriate.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.