How to avoid very large output rasters using grass algorithms

Question

I use r.cost.points from the GRASS algorithms in QGIS to create a accumulated travel costs map that is based on a friction surface raster. My problem is that the output file is more than 20x bigger than the input.

The input file is a raster in .tif format and 5MB filesize. The output file is a .tif as well but filesize increases to 110MB.

I don't really understand this increase, because both have the same extend, resolution, Data Type (Float64) and Filetype.

The stored values differ: while the friction map contains only small integer values that inform the travel costs to cross one gridcell (ranging from 1-36), the output map contains the accumulated travel cost values (ranging from 252-46677.400562602). But still both are Float64 so they should be comparable in terms of space requirements AFAIK.

Comparing both Metadata Properties the only difference is the No Data Value which is in case of the input file -1.7e+308 and in case of the output file nan...

My main problem now is that I want to process an input file of 10GB which causes space problems while the output is created... Can anybody explain why this problem occurs and how I can avoid these increases?

You can test this on your computer with this sample data here as the friction surface and here the source layer containing the destinations.

edits i also tried it with a rasterized version of the source layer but this still creates a File of 98MB.

I use the GUI but the command looks in the console like this.

0   GRASS GIS 7 execution commands
        g.proj -c proj4="+proj=eqc +lat_ts=0 +lat_0=0 +lon_0=-54 +x_0=0 +y_0=0 +ellps=WGS84 +units=m +no_defs"
        r.external input="/home/user/R/R-projects/3_accessibility_map/output/toy_data/2_friction/friction_1.tif" band=1 output=tmp1480016848937 --overwrite -o
        r.external input="/home/user/R/R-projects/3_accessibility_map/output/toy_data/1_rasterized/towns2.tif" band=1 output=tmp1480016848938 --overwrite -o
        g.region n=-1340309.99994 s=-1451939.99994 e=-555477.497208 w=-667107.497208 res=30.0
        r.cost  input=tmp1480016848937 start_raster=tmp1480016848938 -n max_cost="0" null_cost="0" memory="4000" output=output5ae5c0b98f304e92ada91772b870452b --overwrite
        g.region raster=output5ae5c0b98f304e92ada91772b870452b
        r.out.gdal --overwrite -c createopt="TFW=YES,COMPRESS=LZW" input=output5ae5c0b98f304e92ada91772b870452b output="/home/user/Desktop/test2.tif"

ant the first lines of the output

2016-11-24T20:47:51 0   GRASS GIS 7 execution console output
        Cleaning up temporary files...

        Starting GRASS GIS...

        Executing '/home/user/.qgis2//processing/grass7_batch_job.sh' ...

        WARNING: Datum <unknown> not recognised by GRASS and no parameters found

        Default region was updated to the new projection, but if you have multiple mapsets `g.region -d` should be run in each to update the region from the default

Answer 1

Okay. I played around with this dataset. I could identify the following factors that influence raster filesize to different degrees:

1. Geographic extent
2. Resolution
3. Compression when writing out the Gtiff
4. Length of stored values
5. Diversity of stored values

To find this out what factors impact the filesize most, I imported and exported the output.tif (110MB) in R with the Rgdal package. The file was not compressed, even though compression was set when I created this file with r.cost.points in GRASS (seems to be a bug).

First I tried different compressions:

    r.acccost<-raster("output/toy_data/3_acc_cost/acccost_grass_rcost_1.tif")
    writeRaster(r.acccost,"output/toy_data/test1.tif",options=c("COMPRESS=LZMA"),overwrite=T)
    writeRaster(r.acccost,"output/toy_data/test2.tif",options=c("COMPRESS=LZW"),overwrite=T)
    writeRaster(r.acccost,"output/toy_data/test3.tif",options=c("COMPRESS=DEFLATE"),overwrite=T)
    writeRaster(r.acccost,"output/toy_data/test4.tif",options=c("COMPRESS=DEFLATE","ZLEVEL=9"),overwrite=T)

The applied compressions reduced my filesize between 5-30% depending on the applied compression method:

• LZW compressed: 104 MB instead of 110
• LZMA compressed: 77 MB instead of 110
• DEFLATE compressed: 75 MB instead of 110 with both, ZLEVEL 6 (default) and 9.

This is not enough for me, even if I would be able to apply the compression method when executing the r.cost.points algorithm.

Than I converted all values in the Raster to Integer Values which reduces the length of the stored numbers. This procedure reduced my filesize to 39MB. Setting options="NBITS=5" to reduce the bitsize did not influence considerably the results.

    r.acccost<-setValues(r.acccost,as.integer(getValues(r.acccost)))
    writeRaster(r.acccost,"output/toy_data/test5.tif",overwrite=T)

The last thing I did was to reduce the diversity of numbers in my raster. I did so by reclassifying values above the median to 1 and below or equal the median to 2. This procedure reduced my filesize to only 3.2 MB (!)

    r.acccost3<-setValues(r.acccost,ifelse(getValues(r.acccost)>median(getValues(r.acccost),na.rm=T),1,2))
    writeRaster(r.acccost3,"output/toy_data/test6.tif",overwrite=T)

I don't know how rasters work internally but this explains also why the size of my input map in r.cost.points, with relatively few values, was only 5MB.

Summarizing:

I can neither change the extent of the raster nor the length and diversity of created values when using r.cost.points. One option could be to set a maximum value for travelcosts (max_cost=VALUE) when creating the map. However this creates an incomplete dataset. It would be very desirable to be able to influence the length or diversity of created values, but this is so far not possible (see options of r.cost). The later would be desirable for a lot of algorithms when one works with big rasters that are not sub-settable due to the nature of the operation.

I could try to make the compression work which would require more debugging and might be time-consuming. Furthermore I can decrease the resolution which is currently 30meters and probably the best way to start.

For everbody who already has a raster created and only seeks to decrese filesize: besides the extent and resolution you can also look at: Compression, length and diversity of stored values. The diversity has probably the biggest impact and might be easily reduced if you reclassify your raster... which reduces the quality of the data but also reduces considerably the filesize. If it doesnt hurt to much, e.g. 50 classes are way better than 10000 unique values when it comes down to filesize.