Why is gdal_merge resulting in massive output file?

Question

When using gdal merge to merge two geotiff files about 20 and 60 MB in size, the output files end up on the order of tens of GB. I have tried a variety of compression types, but they either fail or result in a massive output file.

Compression types I've tried:

• Packbits
• Deflate
• JPEG
• LZW

Most if not all fail if the BIGTIFF=YES option is not included. When the BIGTIFF=YES option is included, the output files are much too large - even to the point where my 32GB memory laptop begins to have memory errors.

I also tried to force a Bytes output as the two input files seem to be in a Bytes format based on the results of gdal_info. That too did not achieve any better results.

Also note, one of my original geotiffs had very high resolution, but I used gdal_translate with -outsize at 20% to reduce its size before performing a gdal_merge.

Here is the gdal info of geotiff input 1:

Driver: GTiff/GeoTIFF
Files: .../1162020Ortho_Orthomosaic_export_WedNov11_smalltif.tif
Size is 3302, 1561
Coordinate System is:
PROJCS["WGS 84 / Pseudo-Mercator",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Mercator_1SP"],
    PARAMETER["central_meridian",0],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["X",EAST],
    AXIS["Y",NORTH],
    EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs"],
    AUTHORITY["EPSG","3857"]]
Origin = (-11713066.894992373883724,4823300.694965585134923)
Pixel Size = (0.373272889490753,-0.373371134320608)
Metadata:
  acquisitionEndDate=2020-11-06T18:35:00+00:00
  acquisitionStartDate=2020-11-06T18:05:00+00:00
  AREA_OR_POINT=Area
  isCalibrated=False
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (-11713066.895, 4823300.695) (105d13'12.97"W, 39d42'21.36"N)
Lower Left  (-11713066.895, 4822717.863) (105d13'12.97"W, 39d42' 6.85"N)
Upper Right (-11711834.348, 4823300.695) (105d12'33.11"W, 39d42'21.36"N)
Lower Right (-11711834.348, 4822717.863) (105d12'33.11"W, 39d42' 6.85"N)
Center      (-11712450.621, 4823009.279) (105d12'53.04"W, 39d42'14.10"N)
Band 1 Block=3302x1 Type=Byte, ColorInterp=Red
  Mask Flags: PER_DATASET ALPHA 
Band 2 Block=3302x1 Type=Byte, ColorInterp=Green
  Mask Flags: PER_DATASET ALPHA 
Band 3 Block=3302x1 Type=Byte, ColorInterp=Blue
  Mask Flags: PER_DATASET ALPHA 
Band 4 Block=3302x1 Type=Byte, ColorInterp=Alpha

Here is the gdal info of geotiff input 2:

Driver: GTiff/GeoTIFF
Files: .../google_background_20200811.tif
Size is 5237, 3249
Coordinate System is:
PROJCS["NAD83 / Colorado Central (ftUS)",
    GEOGCS["NAD83",
        DATUM["North_American_Datum_1983",
            SPHEROID["GRS 1980",6378137,298.257222101,
                AUTHORITY["EPSG","7019"]],
            TOWGS84[0,0,0,0,0,0,0],
            AUTHORITY["EPSG","6269"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4269"]],
    PROJECTION["Lambert_Conformal_Conic_2SP"],
    PARAMETER["standard_parallel_1",39.75],
    PARAMETER["standard_parallel_2",38.45],
    PARAMETER["latitude_of_origin",37.83333333333334],
    PARAMETER["central_meridian",-105.5],
    PARAMETER["false_easting",3000000],
    PARAMETER["false_northing",1000000],
    UNIT["US survey foot",0.3048006096012192,
        AUTHORITY["EPSG","9003"]],
    AXIS["X",EAST],
    AXIS["Y",NORTH],
    AUTHORITY["EPSG","2232"]]
Origin = (3078185.035587466321886,1683902.833827972644940)
Pixel Size = (1.000014190675706,-1.000076850601459)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  ( 3078185.036, 1683902.834) (105d13'19.55"W, 39d42'38.83"N)
Lower Left  ( 3078185.036, 1680653.584) (105d13'19.67"W, 39d42' 6.72"N)
Upper Right ( 3083422.110, 1683902.834) (105d12'12.53"W, 39d42'38.67"N)
Lower Right ( 3083422.110, 1680653.584) (105d12'12.67"W, 39d42' 6.56"N)
Center      ( 3080803.573, 1682278.209) (105d12'46.11"W, 39d42'22.69"N)
Band 1 Block=5237x1 Type=Byte, ColorInterp=Red
  Mask Flags: PER_DATASET ALPHA 
Band 2 Block=5237x1 Type=Byte, ColorInterp=Green
  Mask Flags: PER_DATASET ALPHA 
Band 3 Block=5237x1 Type=Byte, ColorInterp=Blue
  Mask Flags: PER_DATASET ALPHA 
Band 4 Block=5237x1 Type=Byte, ColorInterp=Alpha

Example of inputs for gdal_merge with DEFLATE:

gdal_merge.bat -n 0 -a_nodata 0 -co COMPRESS=DEFLATE -co PREDICTOR=2 -co ZLEVEL=9 -co BIGTIFF=YES -of GTiff -o ".../Orthomosaic_Merge_20201120.tif" "...\\1162020Ortho_Orthomosaic_export_WedNov11_smalltif.tif" "...\\google_background_20200811.tif"

Answer 1

I am not sure what happens but your source images are not suitable for the tool and the end result is therefore undefined.

From the documentation https://gdal.org/programs/gdal_merge.html

This utility will automatically mosaic a set of images. All the images must be in the same coordinate system and have a matching number of bands, but they may be overlapping, and at different resolutions. In areas of overlap, the last image will be copied over earlier ones.

Notice All the images must be in the same coordinate system. Your images are in different systems, one in EPSG:3857 and another in EPSG:2232.

EDIT

This is just a guess, but it might be that gdal_merge considers that all images share the coordinate system of the first image. Then the extents of the merger image would cover the area from Upper Left corner of the first image till the Lower Right corner of the second image that would make the image huge

Upper Left  (-11713066.895, 4823300.695)
Lower Right ( 3083422.110, 1680653.584)

The gdalinfo report from Orthomosaic_Merge_20201120.tif could proof the theory right or wrong.