Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I'm having trouble locating any discussion or comparative benchmarking of different raster file formats (e.g., for use in data analysis in R). Does anybody have any insight into why particular formats may be faster or slower? Or should the differences be minimal?

Specifically, I'm interested in if converting a raster (e.g., a GEOTIFF file) to a different format (e.g, a netCDF) is ever worthwhile for the purpose of speeding read/write and other operations.

Thanks in advance!

Best, George

share|improve this question
2  
This question is relevant to GIS, but I suspect you're more likely to get answers on SO, which has a strong subcommunity of R experts. If you don't get an answer quickly, please just flag this question and a moderator will migrate it for you. –  whuber Sep 2 '11 at 15:52

5 Answers 5

Here's an old blog article of mine looking at the file size and access time of the formats. I didn't investigate the write speed, only the access time. I'd say they would probably be directly related, but wouldn't be able to vouch for it.

Article Summary: It seems that Packbits gives you the best access times (at expense of disk space), whereas Deflate gives you intermediate/slow access times for intermediate/small files. Also, you can test access times more empirically by creating thumbnails of various sizes and timing how long it takes. Example command: time gdal_translate -outsize <thumbnail dimensions> -of GTiff <compressed image file> <thumbnail file>

Assuming that the only thing relevant to R in this case is how quickly it can read data from the file, just as any other process would, then that should give you a good indication.

share|improve this answer
    
+1 for the linked article, but the important info is offsite and will be lost to us if that page ever goes down or moves. I suggest giving a summary conclusion of the article so that in the event the page is not available, even momentarily, readers have something to work with for future research and thinking. Thanks! –  matt wilkie Sep 6 '11 at 19:22
    
Fair enough! It seems that Packbits gives you the best access times (at expense of disk space), whereas Deflate gives you intermediate/slow access times for intermediate/small files. Also, you can test access times more empirically by creating thumbnails of various sizes and timing how long it takes. Example command: "time gdal_translate -outsize <thumbnail dimensions> -of GTiff <compressed image file> <thumbnail file>" –  R Thiede Sep 7 '11 at 6:21
1  
thanks! I folded the summary into the answer itself, so it's more self contained (see edit link at bottom left of each answer/question). –  matt wilkie Sep 7 '11 at 18:57
    
Oh, right. Thanks ^.^; –  R Thiede Sep 8 '11 at 8:49

Maybe it is really not a question of which raster image format has better opening benchmarks--rather which raster image formats are most efficient raster source formats for opening and reading as input into an R numerical array. And subsequently--what is the most efficient output format from R assuming you'd be outputting results back to raster.

Either way, if you're going to work with raster in R you will likely be using the rgdal and R ncdf packages to supplement what is contained in the R raster package. With principal reliance on the gdalwarp command. Need to work out format dependencies there to make your raster choice. You'll find a fair bit of coverage on SO and various OSGEO and R forums/blogs/wiki.

But as this is a GIS forum where Python use is in relative ascendency, I'll note that there are advantages to working with raster data in a Python numpy array, with similar dependence on the gdal libraries for raster loading, conversion and export. Some folks find memory management and code structure in Python preferable over native R--perhaps take a look at RPy2 or PypeR as either may be appropriate for your analysis use.

share|improve this answer
    
For manipulating netCDF data (raster source or otherwise) in R, here are links to two R CRAN hosted netCDF packages, ncdf4 -- cran.r-project.org/web/packages/ncdf4/index.html and RNetCDF -- cran.r-project.org/web/packages/RNetCDF/index.html –  V Stuart Foote Sep 4 '11 at 15:17

For read/write operations, you can test the speed of those operations using system.time(). Here are some results from loading a DEM file in R (Raster package) translated into four formats (ASCII, IMG and TIF with no compression and Deflate). For example, on a ~26MB raster:

> system.time(dem <- raster("~/workspace/TEMP/kideporegion.img"))
 user  system elapsed 
 0.154   0.002   0.157 

'Elapsed' gives the total time (seconds) taken for the operation. Running the operations 10 times each and looking at mean elapsed time:

              mean   diff
ASC         0.2237 0.3317
IMG         0.1544 0.0318
tif-Deflate 0.1510 0.0099
tif-none    0.1495 0.0000
tif-pack    0.1513 0.0118

TIFF with no compression is the fastest ... followed very closely by Deflate (0.1% slower) and TIFF-Packbits (1.8% slower), then IMG (3.2% slower) and ASC (33% slower). (This is on a Macbook Pro 2.4 GHz with an SSD, so fast disk operations)

This is simply to load the files, not manipulate them.

share|improve this answer

I read a couple of pages on this several years ago and have since used tiff, packbits, tiled with geotiff header and been happy.
arcpad team article

wiki

But after reading the folowwoing will reconsider and maybe use the deflate variety.

Arcpad site

share|improve this answer

The way you think about these kinds of problems is in terms of how your application accesses your file, vs. how the data is laid out in your file. The idea is that if you can access your data sequentially, it will be much more efficient than if you access it randomly.

GeoTIFF is a collection of 2D "images" or arrays. NetCDF is a general purpose storage for multidimensional arrays. But if you store the arrays in the same way in netCDF as they are in GeoTIFF, you will get the same performance, more or less.

One can also rearrange the data in netCDF, so in principle could optimize for your reading patterns. My guess is that most GIS application are optimized for the GeoTIFF 2D layout, so theres not much to gain by rearranging.

Finally, Id say it really only matters when you have very large files, at least tens of megabytes.

share|improve this answer
    
+1 for the point that random access, or read of arbitrary location, is very different from a sequential one after the other until the whole file is done read. I may be off base, but I think Geotiff also supports tiled-storage and arbitrary access, it's just that by strip/row is the most common and widely supported. Also these days "very large files" in GIS tend to be in the multi GB range. ;-) –  matt wilkie Dec 8 '11 at 22:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.