1

I am using rasterio to create a stack of eight bands and each individual band is about 1GB, but when it's stacking them the output (with a lzw compression) is a tif that's over 53GB! This seems huge to me. Anyone know if this is normal?

def stack_bands(list_of_rasterio_bands, temp_dir, outputname, output_ext):
    # this will stack bands in the order they are in the list, returns a rasterio object of what was created
    print("[INFO] creating {}".format(outputname))

    composite_path = os.path.join(temp_dir, outputname+output_ext)

    arr_stack = []
    for band in list_of_rasterio_bands:
        band_arr = band.read(1, masked=True)
        arr_stack.append(band_arr)

    out_meta = list_of_rasterio_bands[0].meta.copy()
    out_meta.update({"count": len(list_of_rasterio_bands),
                     "nodata": -10000,
                     "compress": 'lzw'})

    with rasterio.open(composite_path, "w", **out_meta, BIGTIFF="IF_SAFER") as dest:
        for band_nr, src in enumerate(arr_stack, start=1):
            dest.write(src, band_nr)
    print("[INFO] {} saved".format(outputname))

    return rasterio.open(composite_path)
  • What compression is used in the source data? Adding data into compressed tiff is also sometimes inefficient. Test that by creating a copy of your 8-band image with gdal_translate. – user30184 May 27 at 13:51
  • the separate bands also each have an lzw compression on them – Emtomp May 27 at 14:01
  • What does gdalinfo say about the compression of your output? Are you sure you are stacking the data correctly? Have you tried using a predictor to tune the compression? Is there any debugging output in your terminal? – bugmenot123 May 27 at 15:34
  • Please try to make the copy with gdal_translate. Check also image sizes in pixels (width x height) and pixel size. – user30184 May 27 at 17:49
  • What happens with masked=False? – snowman2 May 30 at 19:00
0

Check what is the resolution of your data (Uint8, Uint16, Float32, etc). If the original files are Uint8 and when passing through your function they are converted to Float32 (some libraries do that), that would explain such huge file.

Run gdalinfo on the 1 GB rasters and see the band type. Then check the type the band when opened with rasterio io (band_arr inside the for loop).

Hope that helps.

J.

| improve this answer | |
  • gdalinfo on one of the single bands: Size is 38275, 45524 Image Structure Metadata: COMPRESSION=LZW INTERLEAVE=BAND Band 1 Block=38275x1 Type=Int16, ColorInterp=Gray NoData Value=-10000 – Emtomp Jun 3 at 20:09
  • gdalinfo on the stacked raster: Size is 38275, 45524 Image Structure Metadata: COMPRESSION=LZW INTERLEAVE=PIXEL. And each band in the stack has this information: Band 1 Block=38275x1 Type=Int16, ColorInterp=Gray NoData Value=-10000, Band 2 Block=38275x1 Type=Int16, ColorInterp=Undefined NoData Value=-10000, Band3 - Band8 same as band2 's info – Emtomp Jun 3 at 20:10
  • and finally the rasterio .meta info on the stacked raster is {'driver': 'GTiff', 'dtype': 'int16', 'nodata': -10000.0, 'width': 38275, 'height': 45524, 'count': 8, 'crs': CRS.from_epsg(32759), 'transform': Affine(0.5, 0.0, 564791.5, 0.0, -0.5, 5152195.0)} – Emtomp Jun 3 at 20:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.