2

I have a raster file (.tif) that contains integer values from 1 to 31. I have realized that the values are stored as Float32 with NoData Value=-3.39999999999999996e+38 (see raster information below). I think that the optimal method to store the data is as unsigned 8 bit integer (QGIS Byte) as there are no decimal values and the maximum value is below 255. I have used the Translate (convert format) tool available from GDAL to change the data type from Float32 to Byte (following these instructions). However, I have run into two issues:

  1. The resulting file is BIGGER than my original file (from 31MB to 268MB). I noticed that the original file had COMPRESSION=LZW and tried to use various compression formats under Profile (i.e. default, no compression, Low compression, High compress, JPEG compression) but I was not successful in reducing file size while maintaining data quality (e.g. using JPEGwith 100 resulted in some values being +/-1 of the values they should be and some random additional pixels)
  2. I had difficulty setting the NoData value. It seems that 0 worked in my case but I may have files with 0-values that are meaningful and I wouldn't know how to convert the multidecimal NoData value from the Float to an integer value that is not confused with actual zero values in the dataset.

Metadata of original file

Size is 21768, 12920
Pixel Size = (250.000000000000000,-250.000000000000000)
            Image Structure Metadata:
              COMPRESSION=LZW
                  NoData Value=-3.39999999999999996e+38

UPDATE: According to this related post, the raster size is computed based on the following formula:

The size of a raster is just the product of bit-depth/8, bands, rows, and columns plus header metadata (statistics, etc.)

32/8*   1*  21768*  12920*  =1124970240 Bytes/(1024*1024) ~1072MB
8/8*    1*  21768*  12920*  =281242560  Bytes/(1024*1024) ~268MB

That gives me the size Windows indicates for my 8-bit raster without compression.

These are the options in the dialog box:

enter image description here

  • If the size of the data increases when you convert to BYTE, you are probably not going to be able to make your data much smaller than it originally is. Do you need the image to be an image or could you use a different format? For example, if the most important thing is data size rather than visualization, you could read in the data as an array and then write it to a hdf file. You wouldn't be able to visualize this, but you could still access the values for analysis – Nathan Thomas Mar 29 at 10:17
  • I am using QGIS to play around and investigate data values across my study area -- does that imply "visualization"? Sorry for my naiveté. – user3386170 Mar 29 at 15:34
  • Yes, you will need to visualize the data so storing as a hdf won't help. I'm not sure of a way to do what you want if the compressed raster is smaller than the byte format. Can you resize the image so it is smaller? – Nathan Thomas Mar 30 at 18:55
  • What do you mean by resize? I need to keep all the data intact (number of pixels, resolution) but I want to compress it (pyramids or some such could work as many adjacent pixels have the same value). – user3386170 Mar 31 at 18:42
2
  1. The resulting file is bigger because it has no compression. JPEG compression resulted in image degradation because it is a lossy compression method, LZW and DEFLATE are lossless (input value will always = output value). Lossy compression methods are good for data where filesize is more important than absolute data integrity, such as a stretched aerial photo purely for display.

  2. You can set any value (within your data type range) as NoData, you don't have to use 0. In the example below, I use gdalbuildvrt to change the NoData value from -3.4e+38 to 255 which is within the Byte data range 0-255. (Note the -3.39999999999999996e+38 is just because gdalinfo is printing -3.4e+38 with floating point representation error, your NoData is -3.4e+38 which is the minimum value of the Float32 range)

The options for GDAL Translate (convert format) in the GUI window are somewhat hidden. To use the LZW compression method (or other lossless method), choose Profile : Default and then add a compression method with the big green + and enter manually COMPRESS under Name and LZW under Value.

For full functionality of GDAL Translate, use OSGeo4W Shell in Windows (or a terminal shell in Linux/MacOS). It is opened separately from QGIS Desktop, a different program available from the start-up menu. Prior to running the code below, set the path for files with cd followed by path (e.g. cd "C:/Users/yourname/gis files").

E.g (in Linux/MacOS Bash shell):

# Create an LZW compressed version
gdal_translate -co compress=LZW -co tiled=YES test.tif test_lzw.tif

# Convert NoData to 255
# the first file with .vrt is the virtual output, the second is the input
gdalbuildvrt -srcnodata -3.4E+38 -vrtnodata 255 test.vrt test.tif

# Write out a Byte with LZW compression TIFF file
# the first file with .vrt is the input (from buildvrt), the second is the output
gdal_translate -co compress=LZW -co tiled=YES -ot Byte test.vrt test_byte_lzw.tif

# Write out a Byte with no compression TIFF file
gdal_translate -co tiled=YES -ot Byte test.vrt test_byte.tif

# Check file info
gdalinfo test.tif -stats
gdalinfo test_byte_lzw.tif -stats

# Check file size
du -sh test.tif  # Uncompressed
du -h test_lzw.tif
du -h test_byte.tif  # Uncompressed 
du -h test_byte_lzw.tif  #My file uses random 0-31 values so will always be much bigger compressed than "real" data which compresses better.

Output:

Driver: GTiff/GeoTIFF
Files: test.tif
       test.tif.aux.xml
Size is 21768, 12920
<snip>
Band 1 Block=21768x1 Type=Float32, ColorInterp=Gray
  Min=0.000 Max=20.000 
  Minimum=0.000, Maximum=20.000, Mean=9.999, StdDev=8.165
  NoData Value=-3.39999999999999996e+38

Driver: GTiff/GeoTIFF
Files: test_byte_lzw.tif
Size is 21768, 12920
<snip>
Band 1 Block=256x256 Type=Byte, ColorInterp=Gray
  Minimum=0.000, Maximum=20.000, Mean=9.999, StdDev=8.165
  NoData Value=255

1.1G    test.tif
99M test_lzw.tif
275M    test_byte.tif
64M test_byte_lzw.tif

E.g (in OSGeo4W shell):

REM Create an LZW compressed version
gdal_translate -co compress=LZW -co tiled=YES test.tif test_lzw.tif

REM Convert NoData to 255
REM the first file with .vrt is the virtual output, the second is the input
gdalbuildvrt -srcnodata -3.4E+38 -vrtnodata 255 test.vrt test.tif

REM Write out a Byte with LZW compression TIFF file
REM the first file with .vrt is the input (from buildvrt), the second is the output
gdal_translate -co compress=LZW -co tiled=YES -ot Byte test.vrt test_byte_lzw.tif

REM Write out a Byte with no compression TIFF file
gdal_translate -co tiled=YES -ot Byte test.vrt test_byte.tif

REM Check file info
gdalinfo test.tif -stats
gdalinfo test_byte_lzw.tif -stats

REM Check file size
dir test.tif 
dir test_lzw.tif
dir test_byte.tif
dir test_byte_lzw.tif
  • -a_nodata is a gdal_translate argument. -srcnodata is used in the gdalbuildvrt command as shown in my answer. – user2856 Apr 2 at 2:05
  • Thanks for all your step-by-step help. Sorry for my ignorance. I got it to work but given that I had never used gdal in command line or OSGeo4W Shell (perhaps that is the same thing), there were some additional steps that were not clear for me. I'm going to edit the question to add those additonal steps (mostly from our discussion here) to help other users at my level -- but please re-edit as necessary. – user3386170 Apr 2 at 18:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.